[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:24 dist-test-slave-gqzk krb5kdc[4336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:24 dist-test-slave-gqzk krb5kdc[4336](info): set up 2 sockets
Jun 20 14:30:24 dist-test-slave-gqzk krb5kdc[4336](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:30:25.673785 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:25 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:25.792550 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33105
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:33105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:25.903621 4349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:25.903906 4349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:25.903965 4349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:25.908059 4349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:25.908123 4349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:25.908150 4349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:25.908171 4349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:25.908198 4349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:25.913546 4349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:33105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:25.914628 4349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:25.915624 4349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:25.923979 4355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:25.924006 4357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:25.924386 4354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:25.924543 4349 server_base.cc:1034] running on GCE node
I20260620 14:30:25.925150 4349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:25.926252 4349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:25.927420 4349 hybrid_clock.cc:648] HybridClock initialized: now 1781965825927327 us; error 105 us; skew 500 ppm
Jun 20 14:30:25 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:25.930184 4349 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:25.931102 4349 webserver.cc:460] Webserver started at http://127.4.58.126:33415/ using document root <none> and password file <none>
I20260620 14:30:25.931638 4349 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:25.931696 4349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:25.931900 4349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:25.933975 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "67118b2cf64d4df89fd9a70709776e90"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-gqzk"
server_key: "4258f78bc73bfc970403cf0b848257a9"
server_key_iv: "4cdf721f84df6942283fb0a506cffe65"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.934520 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "67118b2cf64d4df89fd9a70709776e90"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-gqzk"
server_key: "4258f78bc73bfc970403cf0b848257a9"
server_key_iv: "4cdf721f84df6942283fb0a506cffe65"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.938190 4349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:30:25.940577 4364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:25.941859 4349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:25.941975 4349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "67118b2cf64d4df89fd9a70709776e90"
format_stamp: "Formatted at 2026-06-20 14:30:25 on dist-test-slave-gqzk"
server_key: "4258f78bc73bfc970403cf0b848257a9"
server_key_iv: "4cdf721f84df6942283fb0a506cffe65"
server_key_version: "encryptionkey@0"
I20260620 14:30:25.942077 4349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:25.967438 4349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:25.971532 4349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:25.971771 4349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:25.979414 4349 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:33105
I20260620 14:30:25.979481 4416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:33105 every 8 connection(s)
I20260620 14:30:25.980377 4349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:25.983217 4417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:25.988668 4417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: Bootstrap starting.
I20260620 14:30:25.989464 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4349
I20260620 14:30:25.989619 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:25.991256 4329 external_mini_cluster.cc:1428] Setting key 6872dda1ed11d6bd2e29e521aea87d83
I20260620 14:30:25.991837 4417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:25.992798 4417 log.cc:824] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:25.994954 4417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: No bootstrap required, opened a new log
I20260620 14:30:25.998915 4417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } }
I20260620 14:30:25.999153 4417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:25.999246 4417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67118b2cf64d4df89fd9a70709776e90, State: Initialized, Role: FOLLOWER
I20260620 14:30:25.999678 4417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } }
I20260620 14:30:25.999812 4417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:25.999876 4417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:25.999969 4417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:26.001168 4417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } }
I20260620 14:30:26.001549 4417 leader_election.cc:302] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67118b2cf64d4df89fd9a70709776e90; no voters:
I20260620 14:30:26.001927 4417 leader_election.cc:288] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:26.002043 4422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:26.002287 4422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [term 1 LEADER]: Becoming Leader. State: Replica: 67118b2cf64d4df89fd9a70709776e90, State: Running, Role: LEADER
I20260620 14:30:26.002628 4422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } }
I20260620 14:30:26.003291 4417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:26.004336 4423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67118b2cf64d4df89fd9a70709776e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } } }
I20260620 14:30:26.004478 4423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:26.004817 4424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67118b2cf64d4df89fd9a70709776e90. Latest consensus state: current_term: 1 leader_uuid: "67118b2cf64d4df89fd9a70709776e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67118b2cf64d4df89fd9a70709776e90" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33105 } } }
I20260620 14:30:26.004923 4424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:26.008056 4437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:26.008139 4437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:26.008235 4431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:26.009181 4431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:26.016589 4431 catalog_manager.cc:1347] Generated new cluster ID: 6bf90768222341d8b2a6fa7f221b849a
I20260620 14:30:26.016666 4431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:26.018764 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:25.994850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.4.58.126:33105)
0620 14:30:25.995351 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:25.995370 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:25.996707 (+ 1337us) server_negotiation.cc:407] Connection header received
0620 14:30:25.999449 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:25.999466 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:25.999808 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.000143 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.003204 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.004841 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.006638 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.007351 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.007523 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.011709 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.011717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.011736 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.011770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.013919 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.014480 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.014486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.014490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.014628 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.015203 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.015207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.015209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.015654 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.015891 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.017582 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.017876 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":167,"threads_started":1}
I20260620 14:30:26.036278 4431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:26.037670 4431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:26.053071 4431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67118b2cf64d4df89fd9a70709776e90: Generated new TSK 0
I20260620 14:30:26.054005 4431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:26.085847 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:26.199128 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:26.199404 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:26.199447 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:26.203552 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:26.203629 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:26.203711 4443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:26.209163 4443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:26.210189 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:26.211237 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:26.220494 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:26.220506 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.220564 4443 server_base.cc:1034] running on GCE node
W20260620 14:30:26.220506 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.221130 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:26.221745 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:26.222916 4443 hybrid_clock.cc:648] HybridClock initialized: now 1781965826222896 us; error 35 us; skew 500 ppm
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:26.225808 4443 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:26.226662 4443 webserver.cc:460] Webserver started at http://127.4.58.65:42837/ using document root <none> and password file <none>
I20260620 14:30:26.227196 4443 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:26.227262 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:26.227444 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:26.229480 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "eb02c8edf7b94eec811f5c5c35196efe"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3f35b64a28268d7a03900b2a56cf9b1b"
server_key_iv: "b2e1281ddf16e5c497d90d05c9692967"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.230038 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "eb02c8edf7b94eec811f5c5c35196efe"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3f35b64a28268d7a03900b2a56cf9b1b"
server_key_iv: "b2e1281ddf16e5c497d90d05c9692967"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.233855 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 14:30:26.236215 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.237706 4443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:30:26.237823 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "eb02c8edf7b94eec811f5c5c35196efe"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3f35b64a28268d7a03900b2a56cf9b1b"
server_key_iv: "b2e1281ddf16e5c497d90d05c9692967"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.237931 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:26.250226 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:26.253829 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:26.254029 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:26.254673 4443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:26.255645 4443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:26.255702 4443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.255788 4443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:26.255820 4443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.264268 4443 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:36135
I20260620 14:30:26.264297 4571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:36135 every 8 connection(s)
I20260620 14:30:26.265261 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:26.271783 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4443
I20260620 14:30:26.271890 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:26.272128 4329 external_mini_cluster.cc:1428] Setting key 151f9c60020ca75029ba21007ce5b131
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:26.280201 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.267182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:42621 (local address 127.4.58.126:33105)
0620 14:30:26.267354 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:26.267357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:26.268205 (+ 848us) server_negotiation.cc:407] Connection header received
0620 14:30:26.269681 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:26.269685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:26.269774 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.269851 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.271144 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.271675 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.273016 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.273295 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.273379 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.276166 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.276171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.276174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.276205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.277837 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.278500 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.278504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.278506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.278611 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.278947 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.278950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.278953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.279213 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.279353 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.279862 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.280020 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":45}
I20260620 14:30:26.280997 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.267564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33105 (local address 127.4.58.65:42621)
0620 14:30:26.268045 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:26.268092 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:26.269445 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:26.270058 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:26.270067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:26.270461 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:26.270986 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.271001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.271808 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.271811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.272883 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.272894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.273456 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.273460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.273625 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.274295 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:26.274316 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:26.275965 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:26.277965 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.277973 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.277994 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.278373 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.278705 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.278707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.278709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.278860 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.279357 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:26.279363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:26.279623 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 14:30:26.279786 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:30:26.280061 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:26.282269 4572 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33105
I20260620 14:30:26.282518 4572 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:26.283064 4572 heartbeater.cc:511] Master 127.4.58.126:33105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:26.284731 4377 ts_manager.cc:194] Registered new tserver with Master: eb02c8edf7b94eec811f5c5c35196efe (127.4.58.65:36135)
I20260620 14:30:26.285924 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:42621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:26.300482 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:26.412611 4577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:26.412921 4577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:26.412969 4577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:26.417052 4577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:26.417110 4577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:26.417172 4577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:26.422492 4577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:26.423584 4577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:26.424594 4577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:26.434144 4582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.434301 4577 server_base.cc:1034] running on GCE node
W20260620 14:30:26.434121 4585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:26.434103 4583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.434891 4577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:26.435544 4577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:26.436708 4577 hybrid_clock.cc:648] HybridClock initialized: now 1781965826436695 us; error 32 us; skew 500 ppm
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:26.439680 4577 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:26.441044 4577 webserver.cc:460] Webserver started at http://127.4.58.66:34721/ using document root <none> and password file <none>
I20260620 14:30:26.441771 4577 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:26.441864 4577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:26.442183 4577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:26.444516 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "5cfdb0468ee34a95b27edf84db63021b"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "db4d32af6a945cb42fed0dca28403828"
server_key_iv: "27769aacaa171ad05851968f2b6d8c87"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.445165 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "5cfdb0468ee34a95b27edf84db63021b"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "db4d32af6a945cb42fed0dca28403828"
server_key_iv: "27769aacaa171ad05851968f2b6d8c87"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.449330 4577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:26.452059 4592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.453564 4577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:26.453689 4577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "5cfdb0468ee34a95b27edf84db63021b"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "db4d32af6a945cb42fed0dca28403828"
server_key_iv: "27769aacaa171ad05851968f2b6d8c87"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.453797 4577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:26.474954 4577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:26.478288 4577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:26.478478 4577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:26.479094 4577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:26.479936 4577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:26.479984 4577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.480037 4577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:26.480064 4577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.488898 4577 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:42613
I20260620 14:30:26.488921 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:42613 every 8 connection(s)
I20260620 14:30:26.489811 4577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:26.497174 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4577
I20260620 14:30:26.497332 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:26.497581 4329 external_mini_cluster.cc:1428] Setting key f167188540be769e05c727e0026a1202
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:26.504312 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.491669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:55275 (local address 127.4.58.126:33105)
0620 14:30:26.491832 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:26.491837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:26.492773 (+ 936us) server_negotiation.cc:407] Connection header received
0620 14:30:26.494158 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:26.494161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:26.494208 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.494341 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.495570 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.496094 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.497367 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.497627 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.497740 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.500603 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.500606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.500608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.500631 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.502027 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.502646 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.502649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.502651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.502755 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.503120 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.503122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.503123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.503339 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.503432 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.503966 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.504108 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:30:26.505287 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.492106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33105 (local address 127.4.58.66:55275)
0620 14:30:26.492589 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:26.492637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:26.493943 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:26.494505 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:26.494512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:26.494880 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:26.495448 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.495462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.496251 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.496254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.497229 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.497240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.497782 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.497786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.497965 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.498609 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:26.498626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:26.500434 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:26.502142 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.502148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.502166 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.502525 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.502844 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.502848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.502850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.503034 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.503429 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:26.503437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:26.503721 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 14:30:26.503913 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:30:26.504274 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260620 14:30:26.506616 4706 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33105
I20260620 14:30:26.506891 4706 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:26.507395 4706 heartbeater.cc:511] Master 127.4.58.126:33105 requested a full tablet report, sending...
I20260620 14:30:26.508366 4377 ts_manager.cc:194] Registered new tserver with Master: 5cfdb0468ee34a95b27edf84db63021b (127.4.58.66:42613)
I20260620 14:30:26.508999 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:55275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:26.523878 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:26.635285 4711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:26.635567 4711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:26.635622 4711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:26.639768 4711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:26.639832 4711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:26.639894 4711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:26.645303 4711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:26.646350 4711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:26.647320 4711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:26.656582 4716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:26.656589 4717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.656883 4711 server_base.cc:1034] running on GCE node
W20260620 14:30:26.656988 4719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:26.657375 4711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:26.658004 4711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:26.659242 4711 hybrid_clock.cc:648] HybridClock initialized: now 1781965826659128 us; error 65 us; skew 500 ppm
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:26.661893 4711 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:26.662766 4711 webserver.cc:460] Webserver started at http://127.4.58.67:34343/ using document root <none> and password file <none>
I20260620 14:30:26.663288 4711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:26.663350 4711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:26.663519 4711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:26.665556 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "cbede0eae15148d287a7955467fc89a3"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3d3cef45ee4b309af35c2fe09d0252bf"
server_key_iv: "0bb9e43e908ef8f5aba5ea84c978e08c"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.666060 4711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "cbede0eae15148d287a7955467fc89a3"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3d3cef45ee4b309af35c2fe09d0252bf"
server_key_iv: "0bb9e43e908ef8f5aba5ea84c978e08c"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.669793 4711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:26.672144 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.673506 4711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:26.673609 4711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "cbede0eae15148d287a7955467fc89a3"
format_stamp: "Formatted at 2026-06-20 14:30:26 on dist-test-slave-gqzk"
server_key: "3d3cef45ee4b309af35c2fe09d0252bf"
server_key_iv: "0bb9e43e908ef8f5aba5ea84c978e08c"
server_key_version: "encryptionkey@0"
I20260620 14:30:26.673717 4711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:26.689739 4711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:26.692528 4711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:26.692685 4711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:26.693315 4711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:26.694123 4711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:26.694167 4711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.694209 4711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:26.694238 4711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:26.702924 4711 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40895
I20260620 14:30:26.702951 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40895 every 8 connection(s)
I20260620 14:30:26.703827 4711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:26.710247 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4711
I20260620 14:30:26.710346 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:26.710569 4329 external_mini_cluster.cc:1428] Setting key 1716c56fc4611ab0d97605cab7287895
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:26.717031 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.705627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:58641 (local address 127.4.58.126:33105)
0620 14:30:26.705765 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:26.705770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:26.706629 (+ 859us) server_negotiation.cc:407] Connection header received
0620 14:30:26.707924 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:26.707926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:26.707973 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.708045 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.709227 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.709804 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.711024 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.711289 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.711387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.713971 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.713975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.713977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.714002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.715187 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.715641 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.715643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.715645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.715713 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.716061 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.716064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.716065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.716229 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.716304 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.716777 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.716908 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:26.717854 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.706038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33105 (local address 127.4.58.67:58641)
0620 14:30:26.706482 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:26.706524 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:26.707723 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:26.708196 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:26.708203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:26.708579 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:26.709107 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.709121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.709926 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.709929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.710900 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:26.710910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.711429 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.711433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:26.711591 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.712248 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:26.712268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:26.713808 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:26.715290 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.715295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.715308 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.715566 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.715830 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:26.715833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:26.715834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:26.715988 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:26.716304 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:26.716309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:26.716550 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 14:30:26.716702 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:30:26.717029 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260620 14:30:26.718941 4840 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33105
I20260620 14:30:26.719169 4840 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:26.719674 4840 heartbeater.cc:511] Master 127.4.58.126:33105 requested a full tablet report, sending...
I20260620 14:30:26.720631 4377 ts_manager.cc:194] Registered new tserver with Master: cbede0eae15148d287a7955467fc89a3 (127.4.58.67:40895)
I20260620 14:30:26.721202 4377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:58641
I20260620 14:30:26.725085 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:26.750870 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.740810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.4.58.65:36135)
0620 14:30:26.741085 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:26.741102 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:26.741125 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:30:26.741176 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:26.741182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:26.741424 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.741554 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.742193 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.743041 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.744014 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.744437 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.744521 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.747178 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.747189 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.747201 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.747229 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.748941 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.749359 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.749365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.749366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.749505 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.749807 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.749809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.749811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.750079 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.750261 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.750510 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.750672 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
W20260620 14:30:26.751929 4486 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53456
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:26.776659 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:26.768568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.4.58.65:36135)
0620 14:30:26.768727 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:26.768732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:26.768851 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:30:26.769033 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:26.769036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:26.769080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:26.769148 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:26.769859 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.770427 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.771272 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:26.771501 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:26.771591 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:26.773927 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:26.773930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:26.773932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:26.773958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:26.775133 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.775517 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.775520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.775522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.775601 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:26.775850 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:26.775853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:26.775855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:26.776026 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:26.776139 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:26.776384 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:30:26.776508 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:30:26.777735 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4443
I20260620 14:30:26.783542 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4577
I20260620 14:30:26.789896 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4711
I20260620 14:30:26.794801 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4349
2026-06-20T14:30:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2084 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4854](info): set up 2 sockets
Jun 20 14:30:26 dist-test-slave-gqzk krb5kdc[4854](info): commencing operation
krb5kdc: starting...
W20260620 14:30:27.748668 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:27 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:27.867933 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45097
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:27.981668 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:27.981945 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:27.981983 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:27.986019 4867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:27.986078 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:27.986097 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:27.986114 4867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:27.986127 4867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:27.991487 4867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:27.992471 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:27.993465 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:28.001883 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.001888 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.001888 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.002475 4867 server_base.cc:1034] running on GCE node
I20260620 14:30:28.003034 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:28.004209 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:28.005388 4867 hybrid_clock.cc:648] HybridClock initialized: now 1781965828005360 us; error 35 us; skew 500 ppm
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:28.008091 4867 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:28.009017 4867 webserver.cc:460] Webserver started at http://127.4.58.126:36461/ using document root <none> and password file <none>
I20260620 14:30:28.009544 4867 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:28.009599 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:28.009763 4867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:28.011927 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "7274c97a167a4cd79aa4414f262dbf06"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "9ef0bd66fc3fc00d19fea4f8c9d2f1a6"
server_key_iv: "feea94d8e08619e67d488d79c19db331"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.012476 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "7274c97a167a4cd79aa4414f262dbf06"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "9ef0bd66fc3fc00d19fea4f8c9d2f1a6"
server_key_iv: "feea94d8e08619e67d488d79c19db331"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.016233 4867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20260620 14:30:28.018667 4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.019950 4867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:28.020059 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "7274c97a167a4cd79aa4414f262dbf06"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "9ef0bd66fc3fc00d19fea4f8c9d2f1a6"
server_key_iv: "feea94d8e08619e67d488d79c19db331"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.020167 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:28.044139 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:28.047415 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:28.047595 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:28.054591 4867 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:45097
I20260620 14:30:28.054608 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:45097 every 8 connection(s)
I20260620 14:30:28.055555 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:28.058523 4935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:28.064327 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4867
I20260620 14:30:28.064347 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: Bootstrap starting.
I20260620 14:30:28.064445 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:28.064713 4329 external_mini_cluster.cc:1428] Setting key b4da974cd615ea2733d48ed2e3f8db8c
I20260620 14:30:28.067026 4935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:28.067785 4935 log.cc:824] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:28.069916 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: No bootstrap required, opened a new log
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:28.074285 4935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } }
I20260620 14:30:28.074553 4935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:28.074604 4935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7274c97a167a4cd79aa4414f262dbf06, State: Initialized, Role: FOLLOWER
I20260620 14:30:28.075078 4935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } }
I20260620 14:30:28.075207 4935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:28.075270 4935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:28.075361 4935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:28.076241 4935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } }
I20260620 14:30:28.076658 4935 leader_election.cc:302] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7274c97a167a4cd79aa4414f262dbf06; no voters:
I20260620 14:30:28.077067 4935 leader_election.cc:288] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:28.077152 4940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:28.077409 4940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [term 1 LEADER]: Becoming Leader. State: Replica: 7274c97a167a4cd79aa4414f262dbf06, State: Running, Role: LEADER
I20260620 14:30:28.077721 4940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } }
I20260620 14:30:28.078490 4935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:28.080003 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.066087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.4.58.126:45097)
0620 14:30:28.066544 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.066560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.066597 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:28.067272 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.067292 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.067673 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.068015 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.068800 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.069801 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.070940 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.071429 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.071546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.074436 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.074442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.074457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:28.074488 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.076180 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.076673 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.076678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.076683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.076854 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.077161 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.077165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.077166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.077736 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.077972 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.078609 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.078930 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":109,"threads_started":1}
I20260620 14:30:28.080969 4941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7274c97a167a4cd79aa4414f262dbf06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } } }
I20260620 14:30:28.082301 4941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:28.081135 4942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7274c97a167a4cd79aa4414f262dbf06. Latest consensus state: current_term: 1 leader_uuid: "7274c97a167a4cd79aa4414f262dbf06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7274c97a167a4cd79aa4414f262dbf06" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45097 } } }
I20260620 14:30:28.082489 4942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:28.083448 4955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:28.083511 4955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:28.083585 4957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:28.084314 4957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:28.089781 4957 catalog_manager.cc:1347] Generated new cluster ID: aa74296837e248748911c1756a952b6a
I20260620 14:30:28.089859 4957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:28.114180 4957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:28.115188 4957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:28.121518 4957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7274c97a167a4cd79aa4414f262dbf06: Generated new TSK 0
I20260620 14:30:28.122237 4957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:28.157509 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:28.268662 4961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:28.268958 4961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:28.269002 4961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:28.273072 4961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:28.273147 4961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:28.273209 4961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:28.278677 4961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:28.279887 4961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:28.280948 4961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:28.291154 4966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.291155 4967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.291356 4961 server_base.cc:1034] running on GCE node
W20260620 14:30:28.291146 4969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.291976 4961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:28.292646 4961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:28.293839 4961 hybrid_clock.cc:648] HybridClock initialized: now 1781965828293814 us; error 36 us; skew 500 ppm
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:28.296561 4961 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:28.297528 4961 webserver.cc:460] Webserver started at http://127.4.58.65:39397/ using document root <none> and password file <none>
I20260620 14:30:28.298059 4961 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:28.298117 4961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:28.298283 4961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:28.300261 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "a6a1400c0d764291b3aa2efd94c0e88a"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "e63c7a8555592e30343534d7ae385ac5"
server_key_iv: "84175c17bccc71f70fb09d8d1ddcd1f8"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.300810 4961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "a6a1400c0d764291b3aa2efd94c0e88a"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "e63c7a8555592e30343534d7ae385ac5"
server_key_iv: "84175c17bccc71f70fb09d8d1ddcd1f8"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.304396 4961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:28.306958 4976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.308209 4961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:28.308310 4961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "a6a1400c0d764291b3aa2efd94c0e88a"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "e63c7a8555592e30343534d7ae385ac5"
server_key_iv: "84175c17bccc71f70fb09d8d1ddcd1f8"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.308401 4961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:28.331470 4961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:28.334806 4961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:28.334971 4961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:28.335566 4961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:28.336373 4961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:28.336418 4961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.336457 4961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:28.336476 4961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.345074 4961 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:39079
I20260620 14:30:28.345101 5089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:39079 every 8 connection(s)
I20260620 14:30:28.346001 4961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:28.353956 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 4961
I20260620 14:30:28.354063 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:28.354354 4329 external_mini_cluster.cc:1428] Setting key cc1650af7f73041a1e1f1efd841270ef
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:28.359978 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.347851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:34877 (local address 127.4.58.126:45097)
0620 14:30:28.347986 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.347990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.349333 (+ 1343us) server_negotiation.cc:407] Connection header received
0620 14:30:28.350215 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.350218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.350273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.350347 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.351538 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.352085 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.353296 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.353530 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.353618 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.356376 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.356379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.356381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:28.356405 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.357925 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.358433 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.358437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.358439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.358529 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.358854 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.358857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.358858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.359067 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.359160 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.359661 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.359800 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:30:28.360811 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.348240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45097 (local address 127.4.58.65:34877)
0620 14:30:28.349141 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.349186 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.350004 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.350525 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.350533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.350904 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:28.351393 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.351407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.352203 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.352206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.353167 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.353176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.353668 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.353672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.353832 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.354545 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:28.354562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:28.356199 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:28.358042 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.358047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.358060 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.358342 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.358616 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.358618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.358620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.358763 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.359165 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:28.359172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:28.359425 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.359588 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.359906 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":558,"threads_started":1}
I20260620 14:30:28.362076 5090 heartbeater.cc:347] Connected to a master server at 127.4.58.126:45097
I20260620 14:30:28.362365 5090 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:28.362882 5090 heartbeater.cc:511] Master 127.4.58.126:45097 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:28.364460 4899 ts_manager.cc:194] Registered new tserver with Master: a6a1400c0d764291b3aa2efd94c0e88a (127.4.58.65:39079)
I20260620 14:30:28.365592 4899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:34877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:28.380230 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:28.490859 5095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:28.491122 5095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:28.491166 5095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:28.495180 5095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:28.495252 5095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:28.495317 5095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:28.500710 5095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:28.501775 5095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:28.502779 5095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:28.512140 5101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.512122 5100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.512153 5103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.512270 5095 server_base.cc:1034] running on GCE node
I20260620 14:30:28.512916 5095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:28.513573 5095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:28.514732 5095 hybrid_clock.cc:648] HybridClock initialized: now 1781965828514713 us; error 32 us; skew 500 ppm
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:28.517473 5095 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:28.518414 5095 webserver.cc:460] Webserver started at http://127.4.58.66:42811/ using document root <none> and password file <none>
I20260620 14:30:28.518942 5095 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:28.518997 5095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:28.519164 5095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:28.521311 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "7d9a828fc210443198cd406c92443210"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "ac253a678dc7699e1a5d39ac7a907237"
server_key_iv: "5a7b184ce2d00ac6c200e5400745b5c0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.521836 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "7d9a828fc210443198cd406c92443210"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "ac253a678dc7699e1a5d39ac7a907237"
server_key_iv: "5a7b184ce2d00ac6c200e5400745b5c0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.525609 5095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:30:28.528105 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.529587 5095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:30:28.529695 5095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "7d9a828fc210443198cd406c92443210"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "ac253a678dc7699e1a5d39ac7a907237"
server_key_iv: "5a7b184ce2d00ac6c200e5400745b5c0"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.529791 5095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:28.543679 5095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:28.546844 5095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:28.547013 5095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:28.547575 5095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:28.548377 5095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:28.548422 5095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.548460 5095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:28.548477 5095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.556512 5095 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:36041
I20260620 14:30:28.556550 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:36041 every 8 connection(s)
I20260620 14:30:28.557457 5095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:28.566274 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5095
I20260620 14:30:28.566402 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:28.566697 4329 external_mini_cluster.cc:1428] Setting key 860f104da7ed43b43077138650ba581d
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:28.571419 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.559290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:53033 (local address 127.4.58.126:45097)
0620 14:30:28.559437 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.559441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.560255 (+ 814us) server_negotiation.cc:407] Connection header received
0620 14:30:28.561723 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.561727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.561795 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.561868 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.563083 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.563605 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.564741 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.565014 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.565103 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.567726 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.567730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.567733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:28.567758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.569333 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.569809 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.569812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.569848 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.569951 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.570309 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.570311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.570312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.570486 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.570559 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.571178 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.571282 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:30:28.572569 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.559656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45097 (local address 127.4.58.66:53033)
0620 14:30:28.560102 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.560145 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.561500 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.562071 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.562078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.562453 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:28.562938 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.562951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.563726 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.563728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.564621 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.564630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.565166 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.565170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.565332 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.565842 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:28.565858 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:28.567568 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:28.569450 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.569455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.569468 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.569720 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.570045 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.570048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.570050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.570228 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.570573 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:28.570580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:28.570915 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.571121 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.571503 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260620 14:30:28.573876 5224 heartbeater.cc:347] Connected to a master server at 127.4.58.126:45097
I20260620 14:30:28.574122 5224 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:28.574605 5224 heartbeater.cc:511] Master 127.4.58.126:45097 requested a full tablet report, sending...
I20260620 14:30:28.575763 4899 ts_manager.cc:194] Registered new tserver with Master: 7d9a828fc210443198cd406c92443210 (127.4.58.66:36041)
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:28.576565 4899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:53033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:28.596534 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:28.706128 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:28.706394 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:28.706441 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:28.710583 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:28.710639 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:28.710705 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:28.715973 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:45097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:28.717084 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:28.718042 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:28.727519 5235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.727607 5229 server_base.cc:1034] running on GCE node
W20260620 14:30:28.727594 5234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:28.727519 5237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:28.728139 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:28.728741 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:28.729945 5229 hybrid_clock.cc:648] HybridClock initialized: now 1781965828729926 us; error 36 us; skew 500 ppm
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:28.732522 5229 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:28.733363 5229 webserver.cc:460] Webserver started at http://127.4.58.67:44937/ using document root <none> and password file <none>
I20260620 14:30:28.733911 5229 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:28.733968 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:28.734122 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:28.736114 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "db95766ef1194a54820510a805dd5234"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "25a4ddabebabab8143bf44f8b614c830"
server_key_iv: "720869d0431efb08a149c30ba0dd9312"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.736610 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "db95766ef1194a54820510a805dd5234"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "25a4ddabebabab8143bf44f8b614c830"
server_key_iv: "720869d0431efb08a149c30ba0dd9312"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.740089 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:28.742487 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.743717 5229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:28.743821 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "db95766ef1194a54820510a805dd5234"
format_stamp: "Formatted at 2026-06-20 14:30:28 on dist-test-slave-gqzk"
server_key: "25a4ddabebabab8143bf44f8b614c830"
server_key_iv: "720869d0431efb08a149c30ba0dd9312"
server_key_version: "encryptionkey@0"
I20260620 14:30:28.743907 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:28.754262 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:28.757313 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:28.757462 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:28.757994 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:28.758773 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:28.758818 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.758855 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:28.758875 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:28.767329 5229 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40553
I20260620 14:30:28.767377 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40553 every 8 connection(s)
I20260620 14:30:28.768213 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:28.772814 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5229
I20260620 14:30:28.772927 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:28.773167 4329 external_mini_cluster.cc:1428] Setting key 0f8ef781c18181ab69956ed29c3ee21a
Jun 20 14:30:28 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965828, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:28.782950 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.770117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:33629 (local address 127.4.58.126:45097)
0620 14:30:28.770294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.770307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.771155 (+ 848us) server_negotiation.cc:407] Connection header received
0620 14:30:28.772499 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.772502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.772559 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.772656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.774118 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.774832 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.776302 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.776538 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.776636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.779395 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.779399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.779401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:28.779427 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.780873 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.781397 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.781400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.781401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.781483 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.781823 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.781825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.781826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.781999 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.782073 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.782713 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.782822 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:30:28.783819 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.770536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45097 (local address 127.4.58.67:33629)
0620 14:30:28.770999 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.771043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.772299 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.772881 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.772891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.773363 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:28.773966 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.773986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.775010 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.775013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.776155 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.776166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.776689 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.776692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.776910 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.777474 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:28.777491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:28.779146 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:28.780988 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.780994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.781009 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.781301 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.781584 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:28.781587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:28.781588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:28.781744 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:28.782139 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:28.782145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:28.782472 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.782651 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.782949 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260620 14:30:28.785108 5358 heartbeater.cc:347] Connected to a master server at 127.4.58.126:45097
I20260620 14:30:28.785353 5358 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:28.785882 5358 heartbeater.cc:511] Master 127.4.58.126:45097 requested a full tablet report, sending...
I20260620 14:30:28.787050 4898 ts_manager.cc:194] Registered new tserver with Master: db95766ef1194a54820510a805dd5234 (127.4.58.67:40553)
I20260620 14:30:28.787616 4898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:33629
I20260620 14:30:28.788282 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:28.798939 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.791936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52774 (local address 127.4.58.126:45097)
0620 14:30:28.792058 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.792062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.792226 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:30:28.792410 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.792412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.792455 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.792528 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:28.793213 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.793733 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.794547 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.794750 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.794820 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.796197 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:28.796199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:28.796202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:28.796223 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:28.797453 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.797837 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.797840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.797841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.797906 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:28.798112 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:28.798114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:28.798115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:28.798343 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:28.798443 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.798700 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.798807 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:28.803892 4899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:28.806981 4899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:28.821210 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.816029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40553 (local address 127.0.0.1:44646)
0620 14:30:28.816438 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.816450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.816526 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.817297 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.817300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.817315 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:28.817398 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.817405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818736 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.818746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.819919 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.819927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820664 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.820793 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.820835 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.820937 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.821054 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":48,"thread_start_us":84,"threads_started":1}
I20260620 14:30:28.821353 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.815669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.4.58.66:36041)
0620 14:30:28.816323 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.816335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.816359 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:30:28.816435 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.816440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.816654 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.816893 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:28.817405 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818641 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.819892 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820709 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820927 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.821078 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.821131 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.821210 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":92,"threads_started":1}
I20260620 14:30:28.821732 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.815382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.4.58.65:39079)
0620 14:30:28.816666 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.816680 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.816703 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:30:28.816811 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.816815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.817017 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.817164 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:28.817373 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818161 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.819952 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820710 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820953 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.821080 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.821526 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.821619 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":78,"threads_started":1}
I20260620 14:30:28.821835 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.816400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.4.58.67:40553)
0620 14:30:28.816869 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:28.816892 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:28.816912 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:30:28.816983 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:28.816988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:28.817154 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:28.817315 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:28.817510 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818599 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820006 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820554 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820651 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.820846 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:28.821629 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 14:30:28.821720 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":142,"threads_started":1}
I20260620 14:30:28.821233 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.815350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36041 (local address 127.0.0.1:33592)
0620 14:30:28.816003 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.816017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.816105 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.816858 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.816864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.816900 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:28.817038 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.817044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818811 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.818814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.819773 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.819782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820876 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.821011 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.821031 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.821085 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.821140 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":29,"thread_start_us":129,"threads_started":1}
I20260620 14:30:28.821584 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:28.815314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:39079 (local address 127.0.0.1:35168)
0620 14:30:28.815861 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:28.815908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:28.816031 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:28.817151 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:28.817155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:28.817172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:28.817276 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.817283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.818277 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.818280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.819840 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:28.819851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:28.820876 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:28.820880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:28.821322 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:28.821345 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:30:28.821418 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:30:28.821474 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":333,"thread_start_us":86,"threads_started":1}
I20260620 14:30:28.824690 5158 tablet_service.cc:1467] Processing CreateTablet for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2 (DEFAULT_TABLE table=test-table [id=63ae3321d3944d7eb504f6fd65221ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:28.825012 5024 tablet_service.cc:1467] Processing CreateTablet for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2 (DEFAULT_TABLE table=test-table [id=63ae3321d3944d7eb504f6fd65221ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:28.825857 5024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:28.825857 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:28.826649 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2 (DEFAULT_TABLE table=test-table [id=63ae3321d3944d7eb504f6fd65221ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:28.827564 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:28.831737 5375 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Bootstrap starting.
I20260620 14:30:28.833249 5376 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Bootstrap starting.
I20260620 14:30:28.834121 5375 tablet_bootstrap.cc:654] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:28.834779 5377 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Bootstrap starting.
I20260620 14:30:28.835388 5375 log.cc:824] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:28.835947 5376 tablet_bootstrap.cc:654] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:28.836813 5376 log.cc:824] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:28.836943 5377 tablet_bootstrap.cc:654] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:28.837356 5375 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: No bootstrap required, opened a new log
I20260620 14:30:28.837558 5375 ts_tablet_manager.cc:1397] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:30:28.838093 5377 log.cc:824] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:28.838762 5376 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: No bootstrap required, opened a new log
I20260620 14:30:28.838948 5376 ts_tablet_manager.cc:1397] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:30:28.840042 5377 tablet_bootstrap.cc:492] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: No bootstrap required, opened a new log
I20260620 14:30:28.840231 5377 ts_tablet_manager.cc:1397] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 14:30:28.841025 5375 raft_consensus.cc:348] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.841248 5375 raft_consensus.cc:374] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:28.841306 5375 raft_consensus.cc:729] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a1400c0d764291b3aa2efd94c0e88a, State: Initialized, Role: FOLLOWER
I20260620 14:30:28.841723 5375 consensus_queue.cc:260] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.842352 5090 heartbeater.cc:503] Master 127.4.58.126:45097 was elected leader, sending a full tablet report...
I20260620 14:30:28.842700 5375 ts_tablet_manager.cc:1428] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:30:28.842696 5376 raft_consensus.cc:348] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.842969 5376 raft_consensus.cc:374] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:28.843024 5376 raft_consensus.cc:729] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9a828fc210443198cd406c92443210, State: Initialized, Role: FOLLOWER
I20260620 14:30:28.843489 5376 consensus_queue.cc:260] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.843864 5377 raft_consensus.cc:348] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.844106 5377 raft_consensus.cc:374] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:28.844128 5224 heartbeater.cc:503] Master 127.4.58.126:45097 was elected leader, sending a full tablet report...
I20260620 14:30:28.844152 5377 raft_consensus.cc:729] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db95766ef1194a54820510a805dd5234, State: Initialized, Role: FOLLOWER
I20260620 14:30:28.844432 5376 ts_tablet_manager.cc:1428] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:30:28.844595 5377 consensus_queue.cc:260] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:28.845510 5377 ts_tablet_manager.cc:1428] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:30:28.845839 5358 heartbeater.cc:503] Master 127.4.58.126:45097 was elected leader, sending a full tablet report...
W20260620 14:30:28.847257 5091 tablet.cc:2369] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:29.019279 5359 tablet.cc:2369] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:29.058550 5225 tablet.cc:2369] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:29.185631 5381 raft_consensus.cc:482] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:29.185842 5381 raft_consensus.cc:504] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:29.187031 5381 leader_election.cc:288] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9a828fc210443198cd406c92443210 (127.4.58.66:36041), db95766ef1194a54820510a805dd5234 (127.4.58.67:40553)
I20260620 14:30:29.192487 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.187433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:59883 (local address 127.4.58.66:36041)
0620 14:30:29.187556 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.187560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.187824 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:30:29.187968 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.187972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.188038 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.188125 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:29.188504 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.189350 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191172 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.191814 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191935 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.192002 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.192273 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.192357 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:30:29.192488 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.187419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:35427 (local address 127.4.58.67:40553)
0620 14:30:29.187543 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.187547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.187774 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:30:29.187968 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.187972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.188036 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.188124 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:29.188504 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.189387 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191172 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.191863 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.192010 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.192282 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.192357 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:29.192507 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.187315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40553 (local address 127.4.58.65:35427)
0620 14:30:29.187668 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:29.187683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:29.187816 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:29.188147 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:29.188150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:29.188193 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:29.188368 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:29.188375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.189478 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.189481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:29.191027 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:29.191040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.191955 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:29.192119 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.192144 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:30:29.192216 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:29.192314 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":170,"spinlock_wait_cycles":8832,"thread_start_us":97,"threads_started":1}
I20260620 14:30:29.192509 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.187362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36041 (local address 127.4.58.65:59883)
0620 14:30:29.187746 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:29.187756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:29.187827 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:29.188147 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:29.188150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:29.188191 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:29.188381 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:29.188387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.189472 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.189475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:29.191027 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:29.191040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.191942 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.191946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:29.192089 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.192137 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:30:29.192216 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:30:29.192318 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":140,"spinlock_wait_cycles":32512,"thread_start_us":124,"threads_started":1}
I20260620 14:30:29.193235 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f0fbe03f344f5a8ff7e83f9798e8c2" candidate_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db95766ef1194a54820510a805dd5234" is_pre_election: true
I20260620 14:30:29.193393 5178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f0fbe03f344f5a8ff7e83f9798e8c2" candidate_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9a828fc210443198cd406c92443210" is_pre_election: true
I20260620 14:30:29.193563 5309 raft_consensus.cc:2454] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a1400c0d764291b3aa2efd94c0e88a in term 0.
I20260620 14:30:29.193682 5178 raft_consensus.cc:2454] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a1400c0d764291b3aa2efd94c0e88a in term 0.
I20260620 14:30:29.194181 4980 leader_election.cc:302] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a1400c0d764291b3aa2efd94c0e88a, db95766ef1194a54820510a805dd5234; no voters:
I20260620 14:30:29.194473 5381 raft_consensus.cc:2781] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:29.194562 5381 raft_consensus.cc:482] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:29.194604 5381 raft_consensus.cc:3037] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:29.195662 5381 raft_consensus.cc:504] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:29.195996 5381 leader_election.cc:288] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [CANDIDATE]: Term 1 election: Requested vote from peers 7d9a828fc210443198cd406c92443210 (127.4.58.66:36041), db95766ef1194a54820510a805dd5234 (127.4.58.67:40553)
I20260620 14:30:29.196391 5178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f0fbe03f344f5a8ff7e83f9798e8c2" candidate_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9a828fc210443198cd406c92443210"
I20260620 14:30:29.196389 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f0fbe03f344f5a8ff7e83f9798e8c2" candidate_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db95766ef1194a54820510a805dd5234"
I20260620 14:30:29.196516 5178 raft_consensus.cc:3037] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:29.196516 5309 raft_consensus.cc:3037] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:29.197795 5178 raft_consensus.cc:2454] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P 7d9a828fc210443198cd406c92443210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a1400c0d764291b3aa2efd94c0e88a in term 1.
I20260620 14:30:29.197795 5309 raft_consensus.cc:2454] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P db95766ef1194a54820510a805dd5234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a1400c0d764291b3aa2efd94c0e88a in term 1.
I20260620 14:30:29.198108 4980 leader_election.cc:302] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6a1400c0d764291b3aa2efd94c0e88a, db95766ef1194a54820510a805dd5234; no voters:
I20260620 14:30:29.198315 5381 raft_consensus.cc:2781] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:29.198525 5381 raft_consensus.cc:686] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [term 1 LEADER]: Becoming Leader. State: Replica: a6a1400c0d764291b3aa2efd94c0e88a, State: Running, Role: LEADER
I20260620 14:30:29.198871 5381 consensus_queue.cc:237] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } }
I20260620 14:30:29.202026 4897 catalog_manager.cc:5495] T 92f0fbe03f344f5a8ff7e83f9798e8c2 P a6a1400c0d764291b3aa2efd94c0e88a reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a1400c0d764291b3aa2efd94c0e88a (127.4.58.65). New cstate: current_term: 1 leader_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a1400c0d764291b3aa2efd94c0e88a" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d9a828fc210443198cd406c92443210" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db95766ef1194a54820510a805dd5234" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:29.276984 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.268127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.4.58.65:39079)
0620 14:30:29.268294 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.268298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.268433 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:30:29.268556 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.268559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.268608 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.268677 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:29.269422 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.269925 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.270757 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.270979 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.271068 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.273616 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:29.273629 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:29.273645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:29.273672 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:29.275336 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.275768 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.275775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.275776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.275871 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.276087 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.276089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.276090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.276353 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:29.276525 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.276655 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.276794 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:29.302964 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.294202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.4.58.65:39079)
0620 14:30:29.294347 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.294352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.294478 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:30:29.294590 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.294593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.294637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.294708 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:29.295529 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.296054 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.297048 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.297276 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.297362 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.300119 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:29.300122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:29.300124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:29.300150 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:29.301450 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.301840 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.301843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.301844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.301917 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.302106 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.302108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.302109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.302268 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:29.302380 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.302647 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.302775 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[4854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[4854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:29.327522 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:29.319398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.4.58.65:39079)
0620 14:30:29.319546 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:29.319551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:29.319665 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:30:29.319833 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:29.319835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:29.319881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:29.319958 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:29.320655 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.321189 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.322081 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:29.322386 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:29.322474 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:29.324935 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:29.324939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:29.324942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:29.324967 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:29.326283 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.326625 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.326627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.326628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.326713 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:29.326894 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:29.326896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:29.326897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:29.327051 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:29.327158 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:29.327304 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:30:29.327397 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:30:29.328905 5004 tablet_service.cc:2908] Rejecting scan request for tablet 92f0fbe03f344f5a8ff7e83f9798e8c2: Uninitialized: clean time has not yet been initialized
I20260620 14:30:29.330777 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4961
I20260620 14:30:29.336230 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5095
I20260620 14:30:29.341033 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5229
I20260620 14:30:29.345496 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 4867
2026-06-20T14:30:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2550 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[5400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[5400](info): set up 2 sockets
Jun 20 14:30:29 dist-test-slave-gqzk krb5kdc[5400](info): commencing operation
krb5kdc: starting...
W20260620 14:30:30.312506 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:30 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:30.429893 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42269
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:30.540199 5413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:30.540503 5413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:30.540544 5413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:30.544557 5413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:30.544627 5413 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:30:30.544643 5413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:30.544660 5413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:30.544675 5413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:30.544689 5413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:30.550030 5413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:42269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:30.551054 5413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:30.552104 5413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:30.560696 5421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:30.560707 5418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:30.560848 5413 server_base.cc:1034] running on GCE node
W20260620 14:30:30.560707 5419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:30.561563 5413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:30.562721 5413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:30.563892 5413 hybrid_clock.cc:648] HybridClock initialized: now 1781965830563878 us; error 28 us; skew 500 ppm
Jun 20 14:30:30 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:30.576925 5413 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:30.577948 5413 webserver.cc:460] Webserver started at http://127.4.58.126:42933/ using document root <none> and password file <none>
I20260620 14:30:30.578540 5413 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:30.578603 5413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:30.578805 5413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:30.580965 5413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "53781d4be6ae44a3b652e50ba69b29cc"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "a61e5741828bac338cf8b6326352ffb0"
server_key_iv: "5034ef83a45ecf533e8f640af08b23b0"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.581521 5413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "53781d4be6ae44a3b652e50ba69b29cc"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "a61e5741828bac338cf8b6326352ffb0"
server_key_iv: "5034ef83a45ecf533e8f640af08b23b0"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.585364 5413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:30.588042 5428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:30.589434 5413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:30.589571 5413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "53781d4be6ae44a3b652e50ba69b29cc"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "a61e5741828bac338cf8b6326352ffb0"
server_key_iv: "5034ef83a45ecf533e8f640af08b23b0"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.589677 5413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:30.612392 5413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:30.615978 5413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:30.616214 5413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:30.623720 5413 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:42269
I20260620 14:30:30.623742 5490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:42269 every 8 connection(s)
I20260620 14:30:30.624681 5413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:30.626413 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5413
I20260620 14:30:30.626497 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:30.626739 4329 external_mini_cluster.cc:1428] Setting key 8c347d6ba8a18619a6d29c184978d59a
I20260620 14:30:30.627682 5491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:30.633978 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Bootstrap starting.
Jun 20 14:30:30 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:30.636646 5491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:30.637398 5491 log.cc:824] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:30.639330 5491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: No bootstrap required, opened a new log
I20260620 14:30:30.642021 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:30.628216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.4.58.126:42269)
0620 14:30:30.628668 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:30.628685 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:30.628723 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:30.629421 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:30.629437 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:30.629775 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:30.630064 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:30.630943 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.631982 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.633167 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.633689 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.633819 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:30.636773 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:30.636781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:30.636798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:30.636827 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:30.639117 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.639629 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.639634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.639639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.639796 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.640154 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.640157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.640159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.640559 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:30.640723 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:30.641008 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:30:30.641276 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260620 14:30:30.642681 5491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } }
I20260620 14:30:30.642907 5491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:30.642964 5491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53781d4be6ae44a3b652e50ba69b29cc, State: Initialized, Role: FOLLOWER
I20260620 14:30:30.643415 5491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } }
I20260620 14:30:30.643571 5491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:30.643637 5491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:30.643726 5491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:30.644594 5491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } }
I20260620 14:30:30.644927 5491 leader_election.cc:302] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53781d4be6ae44a3b652e50ba69b29cc; no voters:
I20260620 14:30:30.645179 5491 leader_election.cc:288] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:30.645359 5496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:30.645560 5496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [term 1 LEADER]: Becoming Leader. State: Replica: 53781d4be6ae44a3b652e50ba69b29cc, State: Running, Role: LEADER
I20260620 14:30:30.645844 5496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } }
I20260620 14:30:30.646333 5491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:30.647332 5497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53781d4be6ae44a3b652e50ba69b29cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } } }
I20260620 14:30:30.647446 5497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:30.647715 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53781d4be6ae44a3b652e50ba69b29cc. Latest consensus state: current_term: 1 leader_uuid: "53781d4be6ae44a3b652e50ba69b29cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53781d4be6ae44a3b652e50ba69b29cc" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42269 } } }
I20260620 14:30:30.647800 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:30.647907 5505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:30.650696 5505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:30.656076 5505 catalog_manager.cc:1347] Generated new cluster ID: 870090321de642b593cef90a6e6099c6
I20260620 14:30:30.656152 5505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:30.681365 5505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:30.682356 5505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:30.696002 5505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Generated new TSK 0
I20260620 14:30:30.696801 5505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:30.729180 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42269
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:30.841450 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:30.841737 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:30.841784 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:30.845816 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:30.845886 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:30.845927 5517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:30.845960 5517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:30.851595 5517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:30.852690 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:30.853771 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:30.863530 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:30.863656 5517 server_base.cc:1034] running on GCE node
W20260620 14:30:30.863493 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:30.863498 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:30.864264 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:30.864979 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:30.866138 5517 hybrid_clock.cc:648] HybridClock initialized: now 1781965830866118 us; error 36 us; skew 500 ppm
Jun 20 14:30:30 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:30.868783 5517 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:30.869645 5517 webserver.cc:460] Webserver started at http://127.4.58.65:38457/ using document root <none> and password file <none>
I20260620 14:30:30.870151 5517 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:30.870205 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:30.870378 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:30.872365 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "6b02a7b3cd4c4739af0f683ad80992d9"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "9801daffc1f9ca09e508cdd7fd64d823"
server_key_iv: "434202810058d1998a58e9e862213f30"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.872926 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "6b02a7b3cd4c4739af0f683ad80992d9"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "9801daffc1f9ca09e508cdd7fd64d823"
server_key_iv: "434202810058d1998a58e9e862213f30"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.876665 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:30:30.879194 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:30.880599 5517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:30.880714 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "6b02a7b3cd4c4739af0f683ad80992d9"
format_stamp: "Formatted at 2026-06-20 14:30:30 on dist-test-slave-gqzk"
server_key: "9801daffc1f9ca09e508cdd7fd64d823"
server_key_iv: "434202810058d1998a58e9e862213f30"
server_key_version: "encryptionkey@0"
I20260620 14:30:30.880869 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:30.909022 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:30.912313 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:30.912487 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:30.913980 5517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:30.914033 5517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:30.914085 5517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:30.914114 5517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:30:30 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:30.924233 5517 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:42349
I20260620 14:30:30.924604 5647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:42349 every 8 connection(s)
I20260620 14:30:30.925313 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:30.925483 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5517
I20260620 14:30:30.925587 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:30.925907 4329 external_mini_cluster.cc:1428] Setting key b22bf0d5ebd3e023cf22e7fdd74ef209
I20260620 14:30:30.927798 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:30.914914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:60653 (local address 127.4.58.126:42269)
0620 14:30:30.915037 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:30.915041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:30.916154 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 14:30:30.917284 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:30.917288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:30.917398 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:30.917471 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:30.918811 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.919350 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.920504 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.920809 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.920906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:30.923960 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:30.923965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:30.923967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:30.923998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:30.925530 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.926147 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.926151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.926153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.926246 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.926630 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.926633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.926635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.926872 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:30.926967 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:30.927569 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:30:30.927682 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:30.928901 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:30.915483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.65:60653)
0620 14:30:30.915996 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:30.916038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:30.917064 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:30.917661 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:30.917668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:30.918040 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:30.918658 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:30.918674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.919467 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.919471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:30.920380 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:30.920390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.920965 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.920969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:30.921145 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:30.922307 (+ 1162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:30.922329 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:30.923771 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:30.925668 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:30.925675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:30.925691 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:30.926030 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:30.926374 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:30.926377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:30.926379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:30.926539 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:30.926980 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:30.926986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:30.927266 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 14:30:30.927535 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 14:30:30.927874 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260620 14:30:30.930646 5648 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42269
I20260620 14:30:30.930788 5648 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:30.931384 5648 heartbeater.cc:511] Master 127.4.58.126:42269 requested a full tablet report, sending...
I20260620 14:30:30.933144 5444 ts_manager.cc:194] Registered new tserver with Master: 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65:42349)
I20260620 14:30:30.934721 5444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:60653
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:30.942203 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:30.932736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:33871 (local address 127.4.58.126:42269)
0620 14:30:30.932886 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:30.932890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:30.935603 (+ 2713us) server_negotiation.cc:407] Connection header received
0620 14:30:30.935724 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:30.935727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:30.935773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:30.935848 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:30.936429 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.937180 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.937950 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.938161 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.938248 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:30.939298 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:30.939301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:30.939304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:30.939326 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:30.940682 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.941125 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.941127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.941128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.941198 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:30.941454 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:30.941456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:30.941457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:30.941624 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:30.941703 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:30.941980 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:30:30.942088 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:30:30.942263 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:30.932679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.65:33871)
0620 14:30:30.935511 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:30.935525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:30.935633 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:30.935890 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:30.935893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:30.936107 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:30.936238 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:30.936247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.937298 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.937301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:30.937858 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:30.937866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:30.938255 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:30.938257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:30.938319 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:30.938842 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:30.938854 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:30.939160 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:30.940807 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:30.940810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:30.940812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:30.941044 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:30.941269 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:30.941272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:30.941274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:30.941360 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:30.941707 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:30.941710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:30.941813 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:30:30.941917 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:30:30.942061 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2741,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:30.955358 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42269
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:31.063833 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:31.064088 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:31.064132 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:31.068094 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:31.068159 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:31.068199 5657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:31.068230 5657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:31.073467 5657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:31.074504 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:31.075477 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:31.084779 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:31.084832 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:31.084796 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:31.085094 5657 server_base.cc:1034] running on GCE node
I20260620 14:30:31.085574 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:31.086238 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:31.087432 5657 hybrid_clock.cc:648] HybridClock initialized: now 1781965831087417 us; error 24 us; skew 500 ppm
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:31.090065 5657 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:31.090932 5657 webserver.cc:460] Webserver started at http://127.4.58.66:34081/ using document root <none> and password file <none>
I20260620 14:30:31.091461 5657 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:31.091524 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:31.091709 5657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:31.093772 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "13a5a55767374a778bc2ba530a4e24d6"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "5ece4a959057345037042e258fc06cfd"
server_key_iv: "51d2d272e06dfeee272fe800d78d3ab1"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.094298 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "13a5a55767374a778bc2ba530a4e24d6"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "5ece4a959057345037042e258fc06cfd"
server_key_iv: "51d2d272e06dfeee272fe800d78d3ab1"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.097891 5657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:30:31.100227 5672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.101504 5657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:31.101615 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "13a5a55767374a778bc2ba530a4e24d6"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "5ece4a959057345037042e258fc06cfd"
server_key_iv: "51d2d272e06dfeee272fe800d78d3ab1"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.101708 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:31.114878 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:31.117923 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:31.118093 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:31.119534 5657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:31.119586 5657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.119640 5657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:31.119669 5657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:31.129643 5657 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:38655
I20260620 14:30:31.129984 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:38655 every 8 connection(s)
I20260620 14:30:31.130780 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:31.131124 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5657
I20260620 14:30:31.131238 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:31.131551 4329 external_mini_cluster.cc:1428] Setting key 74e460bfba7d1e7a1d2e040fa5ea46d7
I20260620 14:30:31.133201 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.120379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:50231 (local address 127.4.58.126:42269)
0620 14:30:31.120496 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.120499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.121618 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 14:30:31.122624 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.122627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.122688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.122789 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.124290 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.125134 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.126389 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.126720 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.126829 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.129323 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.129327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.129329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.129359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.131039 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.131675 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.131678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.131680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.131771 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.132113 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.132117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.132119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.132327 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.132423 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.132967 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.133086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:30:31.134298 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.120973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.66:50231)
0620 14:30:31.121463 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.121509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.122395 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.122995 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.123005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.123413 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:31.124091 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.124110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.125253 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.125257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.126283 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.126292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.126856 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.126859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.126995 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.127668 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:31.127688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:31.129142 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:31.131177 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.131184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.131199 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.131518 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.131874 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.131878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.131880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.132026 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.132444 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:31.132449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:31.132687 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.132910 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.133240 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":144,"threads_started":1}
I20260620 14:30:31.135697 5788 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42269
I20260620 14:30:31.135862 5788 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:31.136399 5788 heartbeater.cc:511] Master 127.4.58.126:42269 requested a full tablet report, sending...
I20260620 14:30:31.137346 5445 ts_manager.cc:194] Registered new tserver with Master: 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655)
I20260620 14:30:31.138023 5445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:50231
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:30:31.148730 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.137713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.66:48225)
0620 14:30:31.141070 (+ 3357us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.141083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.141235 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.141515 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.141517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.141749 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:31.141848 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.141854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.142640 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.142643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.143448 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.143459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.144017 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.144020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.144095 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.144606 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:31.144621 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:31.145097 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:31.146842 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.146846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.146848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.147088 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.147461 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.147463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.147465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.147547 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.148086 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:31.148090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:31.148202 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.148418 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.148560 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3221,"thread_start_us":94,"threads_started":1}
I20260620 14:30:31.148823 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.137773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:48225 (local address 127.4.58.126:42269)
0620 14:30:31.137876 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.137879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.141170 (+ 3291us) server_negotiation.cc:407] Connection header received
0620 14:30:31.141336 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.141339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.141388 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.141488 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.141952 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.142541 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.143560 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.143881 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.143982 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.145287 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.145290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.145292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.145314 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.146741 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.147213 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.147218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.147219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.147318 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.147630 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.147633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.147635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.147956 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.148071 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.148485 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.148612 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:31.157851 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42269
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:31.266119 5797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:31.266403 5797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:31.266466 5797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:31.270498 5797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:31.270566 5797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:31.270619 5797 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:31.270674 5797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:31.276078 5797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:31.277249 5797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:31.278278 5797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:31.287992 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:31.287997 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:31.288093 5797 server_base.cc:1034] running on GCE node
W20260620 14:30:31.288039 5805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:31.288669 5797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:31.289340 5797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:31.290504 5797 hybrid_clock.cc:648] HybridClock initialized: now 1781965831290488 us; error 30 us; skew 500 ppm
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:31.293170 5797 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:31.294065 5797 webserver.cc:460] Webserver started at http://127.4.58.67:42801/ using document root <none> and password file <none>
I20260620 14:30:31.294625 5797 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:31.294688 5797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:31.294860 5797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:31.296924 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "556e45e656984c049cb7722cabc8e000"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "4248efe8ebb37c97b60f06812e224c8d"
server_key_iv: "efd62e6c821114e9487cb5c667cc5bb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.297446 5797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "556e45e656984c049cb7722cabc8e000"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "4248efe8ebb37c97b60f06812e224c8d"
server_key_iv: "efd62e6c821114e9487cb5c667cc5bb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.301014 5797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:30:31.303380 5812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.304612 5797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:31.304708 5797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "556e45e656984c049cb7722cabc8e000"
format_stamp: "Formatted at 2026-06-20 14:30:31 on dist-test-slave-gqzk"
server_key: "4248efe8ebb37c97b60f06812e224c8d"
server_key_iv: "efd62e6c821114e9487cb5c667cc5bb9"
server_key_version: "encryptionkey@0"
I20260620 14:30:31.304863 5797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:31.316444 5797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:31.319765 5797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:31.319928 5797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:31.321420 5797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:31.321480 5797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.321537 5797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:31.321563 5797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:31.331276 5797 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:45933
I20260620 14:30:31.331362 5927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:45933 every 8 connection(s)
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:31.332369 5797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:31.333720 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 5797
I20260620 14:30:31.333845 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:31.334156 4329 external_mini_cluster.cc:1428] Setting key 6862c5c2c19956bd9c252cab040866a7
I20260620 14:30:31.336406 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.322214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:44865 (local address 127.4.58.126:42269)
0620 14:30:31.322360 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.322366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.323309 (+ 943us) server_negotiation.cc:407] Connection header received
0620 14:30:31.324424 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.324427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.324495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.324578 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.326252 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.327064 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.328730 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.329012 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.329097 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.332343 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.332346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.332348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.332374 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.334025 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.334629 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.334634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.334635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.334724 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.335270 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.335274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.335275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.335502 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.335616 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.336142 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.336271 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:30:31.337431 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.322643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.67:44865)
0620 14:30:31.323116 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.323167 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.324187 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.324847 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.324857 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.325336 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:31.326073 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.326094 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.327185 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.327189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.328584 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.328595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.329162 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.329166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.329374 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.330076 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:31.330098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:31.332057 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:31.334171 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.334176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.334188 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.334508 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.334815 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.334818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.334820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.334999 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.335620 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:31.335628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:31.335873 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.336096 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.336490 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260620 14:30:31.338616 5928 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42269
I20260620 14:30:31.338780 5928 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:31.339339 5928 heartbeater.cc:511] Master 127.4.58.126:42269 requested a full tablet report, sending...
I20260620 14:30:31.340521 5444 ts_manager.cc:194] Registered new tserver with Master: 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933)
I20260620 14:30:31.341202 5444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:44865
I20260620 14:30:31.348155 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:31.348292 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.341234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.4.58.67:51121)
0620 14:30:31.341502 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.341517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.341602 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.341833 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.341836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.342062 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:31.342170 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.342177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.342919 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.342922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.343546 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.343555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.344013 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.344015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.344077 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.344574 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:31.344586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:31.344985 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:31.346552 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.346555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.346557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.346835 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.347146 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.347149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.347151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.347243 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.347697 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:31.347700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:31.347858 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.347966 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.348115 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260620 14:30:31.348300 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.341464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:51121 (local address 127.4.58.126:42269)
0620 14:30:31.341571 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.341575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.341587 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:31.341685 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.341688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.341737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.341801 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.342306 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.342802 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.343688 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.343917 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.343998 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.345125 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.345129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.345131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.345159 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.346443 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.346963 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.346966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.346967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.347045 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.347354 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.347358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.347359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.347598 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.347705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.348032 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.348149 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:31.374162 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.365678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.4.58.126:42269)
0620 14:30:31.365839 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.365842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.365857 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:31.365898 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.365901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.365941 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.366014 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.366759 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.367373 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.368183 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.368425 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.368526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.371060 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.371063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.371065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.371096 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.372459 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.372989 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.372993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.372994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.373083 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.373304 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.373306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.373307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.373503 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.373606 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.373874 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.373990 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:30:31.377225 5444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:31.380097 5444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:31.394220 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.388625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.4.58.66:38655)
0620 14:30:31.389100 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.389113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.389138 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:31.389272 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.389279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.389502 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.390165 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.390317 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.391248 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.392580 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.393481 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.393640 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.393823 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.393970 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.394065 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":138,"threads_started":1}
I20260620 14:30:31.394208 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.388488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38655 (local address 127.0.0.1:32772)
0620 14:30:31.388876 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.388951 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.389085 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.389817 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.389824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.389845 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.390008 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.390016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.391350 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.391353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.392443 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.392454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.393640 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.393643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.393767 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.393812 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.393931 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.394056 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":5,"thread_start_us":81,"threads_started":1}
I20260620 14:30:31.394680 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.389233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42349 (local address 127.0.0.1:52174)
0620 14:30:31.389961 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.389974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.390048 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.390693 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.390696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.390731 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.390825 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.390830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.392125 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.392128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.393202 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.393212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.394326 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.394330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.394411 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.394435 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.394489 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.394555 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260620 14:30:31.394834 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.388842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:45933 (local address 127.0.0.1:44960)
0620 14:30:31.389693 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.389722 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.389803 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.390545 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.390548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.390564 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.390648 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.390654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.392016 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.392020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.393219 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.393230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.394455 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.394459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.394544 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.394566 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.394641 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.394709 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":49,"thread_start_us":92,"threads_started":1}
I20260620 14:30:31.395033 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.389530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.4.58.67:45933)
0620 14:30:31.390089 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.390100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.390124 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:30:31.390202 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.390207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.390419 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.390548 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.390750 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.391883 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.393361 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.394110 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.394220 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.394393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.394705 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.394904 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":89,"threads_started":1}
I20260620 14:30:31.395260 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.389447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.4.58.65:42349)
0620 14:30:31.389830 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.389845 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.390134 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:30:31.390213 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.390219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.390418 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.390560 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.390938 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.392022 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.393318 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.394164 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.394841 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.394998 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.395046 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.395131 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":81,"threads_started":1}
I20260620 14:30:31.397603 5582 tablet_service.cc:1467] Processing CreateTablet for tablet 4062f8f83e2c45dc98449216662757ec (DEFAULT_TABLE table=test-table [id=fcde233e953c4b40a6d14356f830ff18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.397610 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 4062f8f83e2c45dc98449216662757ec (DEFAULT_TABLE table=test-table [id=fcde233e953c4b40a6d14356f830ff18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.397610 5862 tablet_service.cc:1467] Processing CreateTablet for tablet 4062f8f83e2c45dc98449216662757ec (DEFAULT_TABLE table=test-table [id=fcde233e953c4b40a6d14356f830ff18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.398484 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4062f8f83e2c45dc98449216662757ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.398559 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4062f8f83e2c45dc98449216662757ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.398568 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4062f8f83e2c45dc98449216662757ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.405273 5951 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Bootstrap starting.
I20260620 14:30:31.406589 5952 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Bootstrap starting.
I20260620 14:30:31.406613 5953 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Bootstrap starting.
I20260620 14:30:31.407528 5951 tablet_bootstrap.cc:654] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.408303 5951 log.cc:824] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:31.408721 5952 tablet_bootstrap.cc:654] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.408727 5953 tablet_bootstrap.cc:654] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.409564 5952 log.cc:824] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:31.409564 5953 log.cc:824] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:31.410326 5951 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: No bootstrap required, opened a new log
I20260620 14:30:31.410562 5951 ts_tablet_manager.cc:1397] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:30:31.411401 5953 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: No bootstrap required, opened a new log
I20260620 14:30:31.411401 5952 tablet_bootstrap.cc:492] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: No bootstrap required, opened a new log
I20260620 14:30:31.411579 5952 ts_tablet_manager.cc:1397] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:30:31.411579 5953 ts_tablet_manager.cc:1397] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:30:31.413528 5951 raft_consensus.cc:348] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.413761 5951 raft_consensus.cc:374] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.413837 5951 raft_consensus.cc:729] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b02a7b3cd4c4739af0f683ad80992d9, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.414259 5951 consensus_queue.cc:260] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.414870 5648 heartbeater.cc:503] Master 127.4.58.126:42269 was elected leader, sending a full tablet report...
I20260620 14:30:31.415335 5953 raft_consensus.cc:348] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.415335 5952 raft_consensus.cc:348] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.415585 5952 raft_consensus.cc:374] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.415586 5953 raft_consensus.cc:374] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.415639 5953 raft_consensus.cc:729] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a5a55767374a778bc2ba530a4e24d6, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.415684 5952 raft_consensus.cc:729] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556e45e656984c049cb7722cabc8e000, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.416126 5953 consensus_queue.cc:260] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.416126 5952 consensus_queue.cc:260] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.416322 5951 ts_tablet_manager.cc:1428] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:30:31.417100 5953 ts_tablet_manager.cc:1428] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 14:30:31.417410 5788 heartbeater.cc:503] Master 127.4.58.126:42269 was elected leader, sending a full tablet report...
I20260620 14:30:31.417666 5928 heartbeater.cc:503] Master 127.4.58.126:42269 was elected leader, sending a full tablet report...
I20260620 14:30:31.417709 5952 ts_tablet_manager.cc:1428] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260620 14:30:31.426524 5649 tablet.cc:2369] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:31.583519 5929 tablet.cc:2369] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:31.593582 5958 raft_consensus.cc:482] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:31.593804 5958 raft_consensus.cc:504] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.595041 5958 leader_election.cc:288] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65:42349)
I20260620 14:30:31.598557 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.595297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38655 (local address 127.4.58.67:52767)
0620 14:30:31.595449 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.595467 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.595585 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.595841 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.595844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.595862 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.595966 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.595971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.596672 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.596675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.597525 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.597534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.598144 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.598148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.598258 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.598279 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.598351 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.598426 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:30:31.598558 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.595423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:52767 (local address 127.4.58.66:38655)
0620 14:30:31.595559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.595563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.595581 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:30:31.595686 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.595689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.595740 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.595831 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.596064 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.596568 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.597642 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.598039 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.598134 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.598199 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.598374 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.598447 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:30:31.599218 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.595552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42349 (local address 127.4.58.67:48337)
0620 14:30:31.595876 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.595887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.595972 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.596186 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.596188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.596202 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.596634 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.596641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.597381 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.597385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.598123 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.598131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.598843 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.598846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.598936 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.598958 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.599034 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.599095 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260620 14:30:31.599247 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.595674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:48337 (local address 127.4.58.65:42349)
0620 14:30:31.595781 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.595786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.595958 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:30:31.596057 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.596060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.596105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.596165 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.596780 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.597282 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.598247 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.598635 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.598741 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.598796 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.599067 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.599144 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:31.599189 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4062f8f83e2c45dc98449216662757ec" candidate_uuid: "556e45e656984c049cb7722cabc8e000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6" is_pre_election: true
I20260620 14:30:31.599504 5742 raft_consensus.cc:2454] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556e45e656984c049cb7722cabc8e000 in term 0.
I20260620 14:30:31.599776 5602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4062f8f83e2c45dc98449216662757ec" candidate_uuid: "556e45e656984c049cb7722cabc8e000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" is_pre_election: true
I20260620 14:30:31.599975 5813 leader_election.cc:302] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a5a55767374a778bc2ba530a4e24d6, 556e45e656984c049cb7722cabc8e000; no voters:
I20260620 14:30:31.600055 5602 raft_consensus.cc:2454] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556e45e656984c049cb7722cabc8e000 in term 0.
I20260620 14:30:31.600224 5958 raft_consensus.cc:2781] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:31.600350 5958 raft_consensus.cc:482] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:31.600410 5958 raft_consensus.cc:3037] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.601485 5958 raft_consensus.cc:504] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.601781 5958 leader_election.cc:288] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [CANDIDATE]: Term 1 election: Requested vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65:42349)
I20260620 14:30:31.602082 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4062f8f83e2c45dc98449216662757ec" candidate_uuid: "556e45e656984c049cb7722cabc8e000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6"
I20260620 14:30:31.602157 5602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4062f8f83e2c45dc98449216662757ec" candidate_uuid: "556e45e656984c049cb7722cabc8e000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b02a7b3cd4c4739af0f683ad80992d9"
I20260620 14:30:31.602203 5742 raft_consensus.cc:3037] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.602288 5602 raft_consensus.cc:3037] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.603231 5602 raft_consensus.cc:2454] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556e45e656984c049cb7722cabc8e000 in term 1.
I20260620 14:30:31.603252 5742 raft_consensus.cc:2454] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556e45e656984c049cb7722cabc8e000 in term 1.
I20260620 14:30:31.603502 5813 leader_election.cc:302] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a5a55767374a778bc2ba530a4e24d6, 556e45e656984c049cb7722cabc8e000; no voters:
I20260620 14:30:31.603655 5958 raft_consensus.cc:2781] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:31.603850 5958 raft_consensus.cc:686] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 1 LEADER]: Becoming Leader. State: Replica: 556e45e656984c049cb7722cabc8e000, State: Running, Role: LEADER
I20260620 14:30:31.604161 5958 consensus_queue.cc:237] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.607185 5443 catalog_manager.cc:5495] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 reported cstate change: term changed from 0 to 1, leader changed from <none> to 556e45e656984c049cb7722cabc8e000 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "556e45e656984c049cb7722cabc8e000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } health_report { overall_health: HEALTHY } } }
W20260620 14:30:31.632131 5789 tablet.cc:2369] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:31.634752 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.630729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.4.58.67:45933)
0620 14:30:31.630867 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.630870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.630884 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:30:31.631084 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.631087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.631142 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.631224 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:31.631498 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.632050 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.633356 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.633570 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.633659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.633997 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:31.634385 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:31.634467 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.634557 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.634608 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:31.640559 5742 raft_consensus.cc:1261] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Refusing update from remote peer 556e45e656984c049cb7722cabc8e000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:31.640559 5602 raft_consensus.cc:1261] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Refusing update from remote peer 556e45e656984c049cb7722cabc8e000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:31.641196 5958 consensus_queue.cc:1035] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:31.641446 5961 consensus_queue.cc:1035] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:31.649464 5966 mvcc.cc:205] Tried to move back new op lower bound from 7298932046392487936 to 7298932046253322240. Current Snapshot: MvccSnapshot[applied={T|T < 7298932046392487936}]
I20260620 14:30:31.653275 5965 mvcc.cc:205] Tried to move back new op lower bound from 7298932046392487936 to 7298932046253322240. Current Snapshot: MvccSnapshot[applied={T|T < 7298932046392487936 or (T in {7298932046392487936})}]
I20260620 14:30:31.660135 5443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:31.660307 5443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:31.663015 5443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Sending DeleteTablet for 3 replicas of tablet 4062f8f83e2c45dc98449216662757ec
I20260620 14:30:31.663980 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet 4062f8f83e2c45dc98449216662757ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:32772
I20260620 14:30:31.663975 5582 tablet_service.cc:1514] Processing DeleteTablet for tablet 4062f8f83e2c45dc98449216662757ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:52174
I20260620 14:30:31.663967 5862 tablet_service.cc:1514] Processing DeleteTablet for tablet 4062f8f83e2c45dc98449216662757ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44960
I20260620 14:30:31.664464 5975 tablet_replica.cc:330] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: stopping tablet replica
I20260620 14:30:31.664479 5976 tablet_replica.cc:330] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: stopping tablet replica
I20260620 14:30:31.664858 5976 raft_consensus.cc:2229] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:31.664857 5975 raft_consensus.cc:2229] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:31.665208 5976 raft_consensus.cc:2258] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:31.665436 5975 raft_consensus.cc:2258] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:31.665416 5443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:31.665846 5443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:31.667829 5976 ts_tablet_manager.cc:1905] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:31.668141 5975 ts_tablet_manager.cc:1905] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:31.670791 5975 ts_tablet_manager.cc:1918] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:31.670812 5976 ts_tablet_manager.cc:1918] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:31.670905 5975 log.cc:1196] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/4062f8f83e2c45dc98449216662757ec
I20260620 14:30:31.670936 5862 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0bb189fc34857ad0931a1620da007 (DEFAULT_TABLE table=test-table [id=7a85acf2e3b642868c6353eb31b6f15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.671034 5976 log.cc:1196] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/4062f8f83e2c45dc98449216662757ec
I20260620 14:30:31.671217 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0bb189fc34857ad0931a1620da007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.671298 5975 ts_tablet_manager.cc:1939] T 4062f8f83e2c45dc98449216662757ec P 556e45e656984c049cb7722cabc8e000: Deleting consensus metadata
I20260620 14:30:31.671566 5976 ts_tablet_manager.cc:1939] T 4062f8f83e2c45dc98449216662757ec P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting consensus metadata
I20260620 14:30:31.672638 5974 tablet_replica.cc:330] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: stopping tablet replica
I20260620 14:30:31.672986 5974 raft_consensus.cc:2229] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:31.673276 5974 raft_consensus.cc:2258] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:31.673405 5952 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Bootstrap starting.
I20260620 14:30:31.673571 5431 catalog_manager.cc:4841] TS 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933): tablet 4062f8f83e2c45dc98449216662757ec (table test-table [id=fcde233e953c4b40a6d14356f830ff18]) successfully deleted
I20260620 14:30:31.674082 5429 catalog_manager.cc:4841] TS 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65:42349): tablet 4062f8f83e2c45dc98449216662757ec (table test-table [id=fcde233e953c4b40a6d14356f830ff18]) successfully deleted
I20260620 14:30:31.674273 5952 tablet_bootstrap.cc:654] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.674597 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0bb189fc34857ad0931a1620da007 (DEFAULT_TABLE table=test-table [id=7a85acf2e3b642868c6353eb31b6f15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.674840 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0bb189fc34857ad0931a1620da007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.675940 5952 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: No bootstrap required, opened a new log
I20260620 14:30:31.676024 5952 ts_tablet_manager.cc:1397] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:31.676453 5952 raft_consensus.cc:348] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.676575 5952 raft_consensus.cc:374] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.676607 5952 raft_consensus.cc:729] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556e45e656984c049cb7722cabc8e000, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.676721 5952 consensus_queue.cc:260] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.677011 5952 ts_tablet_manager.cc:1428] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:31.676242 5974 ts_tablet_manager.cc:1905] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:31.677301 5953 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Bootstrap starting.
I20260620 14:30:31.677757 5582 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0bb189fc34857ad0931a1620da007 (DEFAULT_TABLE table=test-table [id=7a85acf2e3b642868c6353eb31b6f15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:31.678025 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0bb189fc34857ad0931a1620da007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.679836 5974 ts_tablet_manager.cc:1918] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:31.679926 5974 log.cc:1196] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/4062f8f83e2c45dc98449216662757ec
I20260620 14:30:31.680164 5974 ts_tablet_manager.cc:1939] T 4062f8f83e2c45dc98449216662757ec P 13a5a55767374a778bc2ba530a4e24d6: Deleting consensus metadata
I20260620 14:30:31.680305 5951 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Bootstrap starting.
I20260620 14:30:31.681289 5951 tablet_bootstrap.cc:654] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.681488 5953 tablet_bootstrap.cc:654] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.682253 5429 catalog_manager.cc:4841] TS 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655): tablet 4062f8f83e2c45dc98449216662757ec (table test-table [id=fcde233e953c4b40a6d14356f830ff18]) successfully deleted
I20260620 14:30:31.682991 5953 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: No bootstrap required, opened a new log
I20260620 14:30:31.683079 5953 ts_tablet_manager.cc:1397] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:30:31.683347 5951 tablet_bootstrap.cc:492] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: No bootstrap required, opened a new log
I20260620 14:30:31.683422 5951 ts_tablet_manager.cc:1397] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:30:31.683602 5953 raft_consensus.cc:348] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.683715 5953 raft_consensus.cc:374] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.683746 5953 raft_consensus.cc:729] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a5a55767374a778bc2ba530a4e24d6, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.683835 5951 raft_consensus.cc:348] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.683877 5953 consensus_queue.cc:260] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.683928 5951 raft_consensus.cc:374] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.683955 5951 raft_consensus.cc:729] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b02a7b3cd4c4739af0f683ad80992d9, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.684067 5951 consensus_queue.cc:260] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.684136 5953 ts_tablet_manager.cc:1428] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:31.684314 5951 ts_tablet_manager.cc:1428] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:30:31.706558 5957 raft_consensus.cc:482] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:31.706712 5957 raft_consensus.cc:504] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.708046 5957 leader_election.cc:288] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933)
I20260620 14:30:31.712694 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.708468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:53927 (local address 127.4.58.66:38655)
0620 14:30:31.708581 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.708585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.708732 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:30:31.708929 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.708932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.708977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.709014 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.709418 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.709940 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.711511 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.712125 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.712244 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.712310 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.712500 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.712593 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:31.712699 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.708268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38655 (local address 127.4.58.65:53927)
0620 14:30:31.708612 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.708627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.708796 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.709190 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.709193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.709206 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.709305 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.709310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.710043 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.710046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.711371 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.711384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.712245 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.712248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.712358 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.712380 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.712451 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.712536 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":6,"thread_start_us":84,"threads_started":1}
I20260620 14:30:31.713510 5980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.708268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:45933 (local address 127.4.58.65:56535)
0620 14:30:31.708703 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.708714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.708835 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.709074 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.709077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.709093 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:31.709211 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.709218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.710183 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.710186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.711376 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.711384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.713147 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.713150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.713236 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.713253 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.713315 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.713378 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":48,"thread_start_us":77,"threads_started":1}
I20260620 14:30:31.713534 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.708566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:56535 (local address 127.4.58.67:45933)
0620 14:30:31.708671 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.708675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.708823 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:30:31.708920 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.708923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.708970 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.709050 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:31.709316 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.710076 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.712368 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.713038 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.713142 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.713202 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.713359 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.713432 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:31.713507 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0bb189fc34857ad0931a1620da007" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6" is_pre_election: true
I20260620 14:30:31.713656 5742 raft_consensus.cc:2454] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 0.
I20260620 14:30:31.714036 5533 leader_election.cc:302] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a5a55767374a778bc2ba530a4e24d6, 6b02a7b3cd4c4739af0f683ad80992d9; no voters:
I20260620 14:30:31.714172 5882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0bb189fc34857ad0931a1620da007" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556e45e656984c049cb7722cabc8e000" is_pre_election: true
I20260620 14:30:31.714269 5957 raft_consensus.cc:2781] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:31.714318 5957 raft_consensus.cc:482] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:31.714365 5957 raft_consensus.cc:3037] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.714416 5882 raft_consensus.cc:2454] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 0.
I20260620 14:30:31.715472 5957 raft_consensus.cc:504] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.716121 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0bb189fc34857ad0931a1620da007" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6"
I20260620 14:30:31.716171 5957 leader_election.cc:288] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 election: Requested vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933)
I20260620 14:30:31.716248 5742 raft_consensus.cc:3037] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.716920 5882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0bb189fc34857ad0931a1620da007" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556e45e656984c049cb7722cabc8e000"
I20260620 14:30:31.717028 5882 raft_consensus.cc:3037] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.717355 5742 raft_consensus.cc:2454] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 1.
I20260620 14:30:31.717630 5533 leader_election.cc:302] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a5a55767374a778bc2ba530a4e24d6, 6b02a7b3cd4c4739af0f683ad80992d9; no voters:
I20260620 14:30:31.717886 5957 raft_consensus.cc:2781] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:31.718098 5957 raft_consensus.cc:686] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6b02a7b3cd4c4739af0f683ad80992d9, State: Running, Role: LEADER
I20260620 14:30:31.718366 5882 raft_consensus.cc:2454] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 1.
I20260620 14:30:31.718362 5957 consensus_queue.cc:237] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.720871 5444 catalog_manager.cc:5495] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:30:31 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:31.755795 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.746762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.4.58.126:42269)
0620 14:30:31.746898 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.746902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.747215 (+ 313us) server_negotiation.cc:407] Connection header received
0620 14:30:31.747325 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.747328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.747378 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.747468 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:31.748358 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.749152 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.750000 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.750313 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.750421 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.752697 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:31.752700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:31.752702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:31.752723 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:31.754009 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.754538 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.754541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.754542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.754613 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:31.754903 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:31.754906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:31.754907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:31.755088 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:31.755161 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.755485 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.755649 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":56}
I20260620 14:30:31.755810 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.746753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42269 (local address 127.0.0.1:51372)
0620 14:30:31.747092 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.747111 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.747238 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.747505 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.747507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.748051 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:31.748220 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.748229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.749266 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.749269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.749877 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.749886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.750418 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.750421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.750495 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.751042 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:31.751054 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:31.752544 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:31.754111 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.754121 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.754127 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.754434 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.754709 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:31.754713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:31.754715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:31.754812 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:31.755185 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:31.755190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:31.755325 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.755440 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.755599 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":141,"threads_started":1}
I20260620 14:30:31.759858 5444 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51372:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:30:31.760636 5444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:31.765653 5862 tablet_service.cc:1467] Processing CreateTablet for tablet 029f43570fc6402b89784607f06d184b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa0e0e04ac64fbba711d5254b90abbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:31.765618 5722 tablet_service.cc:1467] Processing CreateTablet for tablet 029f43570fc6402b89784607f06d184b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa0e0e04ac64fbba711d5254b90abbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:31.765643 5582 tablet_service.cc:1467] Processing CreateTablet for tablet 029f43570fc6402b89784607f06d184b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa0e0e04ac64fbba711d5254b90abbc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:31.766026 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f43570fc6402b89784607f06d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.766026 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f43570fc6402b89784607f06d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.766027 5722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029f43570fc6402b89784607f06d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:31.768729 5952 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: Bootstrap starting.
I20260620 14:30:31.769605 5952 tablet_bootstrap.cc:654] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.770237 5953 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: Bootstrap starting.
I20260620 14:30:31.771018 5953 tablet_bootstrap.cc:654] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.772019 5952 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: No bootstrap required, opened a new log
I20260620 14:30:31.772089 5952 ts_tablet_manager.cc:1397] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 14:30:31.772228 5953 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: No bootstrap required, opened a new log
I20260620 14:30:31.772311 5953 ts_tablet_manager.cc:1397] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:31.772694 5952 raft_consensus.cc:348] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.772822 5952 raft_consensus.cc:374] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.772856 5952 raft_consensus.cc:729] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556e45e656984c049cb7722cabc8e000, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.772795 5953 raft_consensus.cc:348] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.772913 5953 raft_consensus.cc:374] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.772945 5953 raft_consensus.cc:729] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a5a55767374a778bc2ba530a4e24d6, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.772965 5952 consensus_queue.cc:260] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.773072 5953 consensus_queue.cc:260] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.773196 5952 ts_tablet_manager.cc:1428] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:31.773346 5959 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.773492 5959 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.773360 5953 ts_tablet_manager.cc:1428] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:31.774470 5969 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.774595 5969 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.776806 5951 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: Bootstrap starting.
I20260620 14:30:31.777779 5951 tablet_bootstrap.cc:654] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:31.778949 5951 tablet_bootstrap.cc:492] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: No bootstrap required, opened a new log
I20260620 14:30:31.779021 5951 ts_tablet_manager.cc:1397] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:30:31.779397 5951 raft_consensus.cc:348] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.779498 5951 raft_consensus.cc:374] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:31.779527 5951 raft_consensus.cc:729] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b02a7b3cd4c4739af0f683ad80992d9, State: Initialized, Role: FOLLOWER
I20260620 14:30:31.779650 5951 consensus_queue.cc:260] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.779940 5951 ts_tablet_manager.cc:1428] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:31.779908 5957 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.780032 5957 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.926781 5957 raft_consensus.cc:482] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:31.926908 5957 raft_consensus.cc:504] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.927294 5957 leader_election.cc:288] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933)
I20260620 14:30:31.927721 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f43570fc6402b89784607f06d184b" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6" is_pre_election: true
I20260620 14:30:31.927877 5742 raft_consensus.cc:2454] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 0.
I20260620 14:30:31.927840 5882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f43570fc6402b89784607f06d184b" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556e45e656984c049cb7722cabc8e000" is_pre_election: true
I20260620 14:30:31.927968 5882 raft_consensus.cc:2454] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 0.
I20260620 14:30:31.928177 5533 leader_election.cc:302] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a5a55767374a778bc2ba530a4e24d6, 6b02a7b3cd4c4739af0f683ad80992d9; no voters:
I20260620 14:30:31.928347 5957 raft_consensus.cc:2781] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:31.928428 5957 raft_consensus.cc:482] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:31.928468 5957 raft_consensus.cc:3037] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.929805 5957 raft_consensus.cc:504] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.930212 5957 leader_election.cc:288] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 election: Requested vote from peers 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655), 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933)
I20260620 14:30:31.930578 5882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f43570fc6402b89784607f06d184b" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556e45e656984c049cb7722cabc8e000"
I20260620 14:30:31.930722 5882 raft_consensus.cc:3037] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.931960 5882 raft_consensus.cc:2454] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 1.
I20260620 14:30:31.932253 5742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029f43570fc6402b89784607f06d184b" candidate_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a5a55767374a778bc2ba530a4e24d6"
I20260620 14:30:31.932396 5742 raft_consensus.cc:3037] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:31.932665 5535 leader_election.cc:302] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556e45e656984c049cb7722cabc8e000, 6b02a7b3cd4c4739af0f683ad80992d9; no voters:
I20260620 14:30:31.932868 5957 raft_consensus.cc:2781] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:31.933058 5957 raft_consensus.cc:686] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6b02a7b3cd4c4739af0f683ad80992d9, State: Running, Role: LEADER
I20260620 14:30:31.933199 5957 consensus_queue.cc:237] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } }
I20260620 14:30:31.933539 5742 raft_consensus.cc:2454] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b02a7b3cd4c4739af0f683ad80992d9 in term 1.
I20260620 14:30:31.934113 5982 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: TxnStatusTablet state changed. Reason: New leader 6b02a7b3cd4c4739af0f683ad80992d9. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.934222 5982 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:31.934511 5994 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:30:31.934665 5994 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:30:31.935338 5443 catalog_manager.cc:5495] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:31.974311 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.968283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42349 (local address 127.0.0.1:52176)
0620 14:30:31.969544 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:31.969560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:31.969695 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:31.970122 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:31.970126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:31.970135 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:31.970256 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.970263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.971096 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.971100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.972053 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:31.972067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.972782 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.972786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:31.972880 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.972949 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:31.973885 (+ 936us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:31.973934 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:30:31.974025 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:30:31.974124 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138}
I20260620 14:30:31.974705 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.968459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.4.58.65:42349)
0620 14:30:31.968596 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.968601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.969621 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 14:30:31.969920 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.969923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.969993 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.970093 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:31.970387 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.970983 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.972203 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.972549 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.972654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.973168 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:31.973790 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:31.973911 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.974512 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.974580 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:31.977911 5882 raft_consensus.cc:1261] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Refusing update from remote peer 6b02a7b3cd4c4739af0f683ad80992d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:31.978405 5742 raft_consensus.cc:1261] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Refusing update from remote peer 6b02a7b3cd4c4739af0f683ad80992d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:31.978500 5982 consensus_queue.cc:1035] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:31.978832 5957 consensus_queue.cc:1035] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:31.980094 5959 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: TxnStatusTablet state changed. Reason: New leader 6b02a7b3cd4c4739af0f683ad80992d9. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.980213 5959 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.980405 5969 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: TxnStatusTablet state changed. Reason: New leader 6b02a7b3cd4c4739af0f683ad80992d9. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.980522 5969 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.982416 5982 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.982534 5982 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:31.982973 5964 mvcc.cc:205] Tried to move back new op lower bound from 7298932047775457280 to 7298932047600013312. Current Snapshot: MvccSnapshot[applied={T|T < 7298932047775457280}]
I20260620 14:30:31.983136 5969 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.983239 5969 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 556e45e656984c049cb7722cabc8e000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.985018 5957 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.985128 5957 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 6b02a7b3cd4c4739af0f683ad80992d9: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:31.987193 5959 tablet_replica.cc:439] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b02a7b3cd4c4739af0f683ad80992d9" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42349 } } peers { permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 } } peers { permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 } } }
I20260620 14:30:31.987306 5959 tablet_replica.cc:442] T 029f43570fc6402b89784607f06d184b P 13a5a55767374a778bc2ba530a4e24d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:31.992595 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:31.989465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.4.58.65:42349)
0620 14:30:31.989575 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:31.989579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:31.989594 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:31.989684 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:31.989687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:31.989734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:31.989815 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:31.989995 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.990778 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.991556 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:31.991899 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:31.991997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:31.992170 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:31.992249 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:31.992319 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:31.992410 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:30:31.992461 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:32.003924 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.000055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42349 (local address 127.4.58.65:33499)
0620 14:30:32.000333 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:32.000344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:32.000448 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:32.000640 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:32.000642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:32.000649 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:32.000794 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:32.000801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.001563 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.001566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:32.002555 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:32.002565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.003139 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.003142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:32.003325 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:32.003376 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:32.003650 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:32.003654 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:32.003695 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 14:30:32.003765 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":19,"thread_start_us":104,"threads_started":1}
I20260620 14:30:32.003932 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.000123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:33499 (local address 127.4.58.65:42349)
0620 14:30:32.000236 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:32.000241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:32.000412 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:30:32.000524 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:32.000527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:32.000564 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:32.000626 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:32.000888 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.001454 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.002675 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.003018 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.003127 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:32.003492 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:32.003579 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:32.003649 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:32.003720 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:30:32.003773 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":38}
I20260620 14:30:32.010948 5742 raft_consensus.cc:1261] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Refusing update from remote peer 6b02a7b3cd4c4739af0f683ad80992d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:32.010948 5882 raft_consensus.cc:1261] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Refusing update from remote peer 6b02a7b3cd4c4739af0f683ad80992d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:32.011442 5999 consensus_queue.cc:1035] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a5a55767374a778bc2ba530a4e24d6" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:32.011605 5957 consensus_queue.cc:1035] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "556e45e656984c049cb7722cabc8e000" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:32.052433 5445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:32.052620 5445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:32.054759 5445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53781d4be6ae44a3b652e50ba69b29cc: Sending DeleteTablet for 3 replicas of tablet 87c0bb189fc34857ad0931a1620da007
I20260620 14:30:32.055457 5722 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0bb189fc34857ad0931a1620da007 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:32 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:32772
I20260620 14:30:32.055471 5581 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0bb189fc34857ad0931a1620da007 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:32 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:52174
I20260620 14:30:32.055491 5862 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0bb189fc34857ad0931a1620da007 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:32 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44960
I20260620 14:30:32.055650 5976 tablet_replica.cc:330] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: stopping tablet replica
I20260620 14:30:32.055696 5974 tablet_replica.cc:330] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: stopping tablet replica
I20260620 14:30:32.055788 5974 raft_consensus.cc:2229] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:32.055774 5976 raft_consensus.cc:2229] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:32.055914 5974 raft_consensus.cc:2258] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:32.056094 5975 tablet_replica.cc:330] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: stopping tablet replica
I20260620 14:30:32.056110 5976 raft_consensus.cc:2258] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:32.056214 5975 raft_consensus.cc:2229] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:32.056334 5975 raft_consensus.cc:2258] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:32.057082 5974 ts_tablet_manager.cc:1905] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:32.057144 5975 ts_tablet_manager.cc:1905] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:32.057230 5976 ts_tablet_manager.cc:1905] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:32.059836 5976 ts_tablet_manager.cc:1918] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:32.059831 5975 ts_tablet_manager.cc:1918] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:32.059900 5976 log.cc:1196] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/87c0bb189fc34857ad0931a1620da007
I20260620 14:30:32.059900 5975 log.cc:1196] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/87c0bb189fc34857ad0931a1620da007
I20260620 14:30:32.059894 5974 ts_tablet_manager.cc:1918] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:32.059966 5974 log.cc:1196] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/87c0bb189fc34857ad0931a1620da007
I20260620 14:30:32.060139 5976 ts_tablet_manager.cc:1939] T 87c0bb189fc34857ad0931a1620da007 P 6b02a7b3cd4c4739af0f683ad80992d9: Deleting consensus metadata
I20260620 14:30:32.060139 5975 ts_tablet_manager.cc:1939] T 87c0bb189fc34857ad0931a1620da007 P 556e45e656984c049cb7722cabc8e000: Deleting consensus metadata
I20260620 14:30:32.060202 5974 ts_tablet_manager.cc:1939] T 87c0bb189fc34857ad0931a1620da007 P 13a5a55767374a778bc2ba530a4e24d6: Deleting consensus metadata
I20260620 14:30:32.061053 5429 catalog_manager.cc:4841] TS 6b02a7b3cd4c4739af0f683ad80992d9 (127.4.58.65:42349): tablet 87c0bb189fc34857ad0931a1620da007 (table test-table [id=7a85acf2e3b642868c6353eb31b6f15b]) successfully deleted
I20260620 14:30:32.061251 5429 catalog_manager.cc:4841] TS 13a5a55767374a778bc2ba530a4e24d6 (127.4.58.66:38655): tablet 87c0bb189fc34857ad0931a1620da007 (table test-table [id=7a85acf2e3b642868c6353eb31b6f15b]) successfully deleted
I20260620 14:30:32.061478 5431 catalog_manager.cc:4841] TS 556e45e656984c049cb7722cabc8e000 (127.4.58.67:45933): tablet 87c0bb189fc34857ad0931a1620da007 (table test-table [id=7a85acf2e3b642868c6353eb31b6f15b]) successfully deleted
Jun 20 14:30:32 dist-test-slave-gqzk krb5kdc[5400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:32.071659 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.060618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.4.58.65:42349)
0620 14:30:32.060742 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:32.060768 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:32.061154 (+ 386us) server_negotiation.cc:407] Connection header received
0620 14:30:32.061324 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:32.061327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:32.061378 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:32.061448 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:32.062479 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.063030 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.064268 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.064501 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.064583 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:32.068140 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:32.068148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:32.068162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:32.068185 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:32.069868 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:32.070340 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:32.070345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:32.070347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:32.070435 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:32.070709 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:32.070711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:32.070712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:32.070983 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:32.071141 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:32.071416 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:30:32.071518 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:30:32.072142 5642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52190
I20260620 14:30:32.081717 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:32.074023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.4.58.126:42269)
0620 14:30:32.074144 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:32.074148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:32.074187 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:30:32.074357 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:32.074360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:32.074405 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:32.074473 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:32.075293 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.075822 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.077008 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:32.077244 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:32.077329 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:32.078853 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:32.078857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:32.078859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:32.078885 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:32.080187 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:32.080625 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:32.080628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:32.080629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:32.080702 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:32.080953 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:32.080955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:32.080956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:32.081132 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:32.081230 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:32.081463 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:30:32.081562 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:30:32.082127 5445 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51386
I20260620 14:30:32.083472 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5517
I20260620 14:30:32.090440 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5657
I20260620 14:30:32.095517 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5797
I20260620 14:30:32.101699 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 5413
2026-06-20T14:30:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2758 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:32 dist-test-slave-gqzk krb5kdc[6012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:32 dist-test-slave-gqzk krb5kdc[6012](info): set up 2 sockets
Jun 20 14:30:32 dist-test-slave-gqzk krb5kdc[6012](info): commencing operation
krb5kdc: starting...
W20260620 14:30:33.072511 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:33.193888 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38205
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:38205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:33.302083 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:33.302348 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:33.302387 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:33.306354 6025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:33.306422 6025 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:30:33.306438 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:33.306455 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:33.306470 6025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:33.306483 6025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:33.311717 6025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:38205
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:33.312774 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:33.313730 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:33.322476 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.322522 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.322517 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.322554 6025 server_base.cc:1034] running on GCE node
I20260620 14:30:33.323333 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:33.324448 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:33.325616 6025 hybrid_clock.cc:648] HybridClock initialized: now 1781965833325564 us; error 65 us; skew 500 ppm
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:33.328351 6025 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:33.329310 6025 webserver.cc:460] Webserver started at http://127.4.58.126:41683/ using document root <none> and password file <none>
I20260620 14:30:33.329856 6025 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:33.329919 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:33.330106 6025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:33.332134 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "f99bb414a34e46ff9a7347540100634c"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "5df48c6dafe1963a7e5ef518999913c6"
server_key_iv: "26a0db9e53b1a77eaf81a0bdc2f23c93"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.332665 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "f99bb414a34e46ff9a7347540100634c"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "5df48c6dafe1963a7e5ef518999913c6"
server_key_iv: "26a0db9e53b1a77eaf81a0bdc2f23c93"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.336217 6025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:33.338717 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.339972 6025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:33.340093 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "f99bb414a34e46ff9a7347540100634c"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "5df48c6dafe1963a7e5ef518999913c6"
server_key_iv: "26a0db9e53b1a77eaf81a0bdc2f23c93"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.340193 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:33.366140 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:33.369519 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:33.369707 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:33.377812 6025 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:38205
I20260620 14:30:33.377821 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:38205 every 8 connection(s)
I20260620 14:30:33.378831 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:33.380126 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6025
I20260620 14:30:33.380230 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:33.380472 4329 external_mini_cluster.cc:1428] Setting key 77dea64785cbbc105474df32b3b339ec
I20260620 14:30:33.382045 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:33.388648 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Bootstrap starting.
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:33.391125 6103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:33.392198 6103 log.cc:824] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:33.394255 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: No bootstrap required, opened a new log
I20260620 14:30:33.394613 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.381800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.4.58.126:38205)
0620 14:30:33.382222 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.382238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.382273 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:30:33.382910 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.382926 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.383256 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.383551 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.384439 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.385490 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.386617 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.387086 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.387205 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.389969 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.389974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.389989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.390020 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.391765 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.392205 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.392211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.392216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.392338 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.392593 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.392596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.392598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.393064 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.393254 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.393527 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.393818 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260620 14:30:33.397712 6103 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } }
I20260620 14:30:33.397956 6103 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:33.398015 6103 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99bb414a34e46ff9a7347540100634c, State: Initialized, Role: FOLLOWER
I20260620 14:30:33.398445 6103 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } }
I20260620 14:30:33.398587 6103 raft_consensus.cc:388] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:33.398653 6103 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:33.398749 6103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:33.399662 6103 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } }
I20260620 14:30:33.399987 6103 leader_election.cc:302] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99bb414a34e46ff9a7347540100634c; no voters:
I20260620 14:30:33.400292 6103 leader_election.cc:288] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:33.400439 6108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:33.400687 6108 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [term 1 LEADER]: Becoming Leader. State: Replica: f99bb414a34e46ff9a7347540100634c, State: Running, Role: LEADER
I20260620 14:30:33.401050 6108 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } }
I20260620 14:30:33.401610 6103 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:33.402606 6109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99bb414a34e46ff9a7347540100634c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } } }
I20260620 14:30:33.402740 6109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:33.403029 6110 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99bb414a34e46ff9a7347540100634c. Latest consensus state: current_term: 1 leader_uuid: "f99bb414a34e46ff9a7347540100634c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99bb414a34e46ff9a7347540100634c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38205 } } }
I20260620 14:30:33.403115 6110 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:33.403216 6117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:33.405987 6117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:33.411392 6117 catalog_manager.cc:1347] Generated new cluster ID: c89f54852962467fad2655059a2de1bb
I20260620 14:30:33.411473 6117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:33.434165 6117 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:33.435451 6117 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:33.442385 6117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Generated new TSK 0
I20260620 14:30:33.443295 6117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:33.475200 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:33.586627 6129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:33.586933 6129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:33.586984 6129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:33.591303 6129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:33.591383 6129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:33.591426 6129 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:30:33.591439 6129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:33.591480 6129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:33.597077 6129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:33.598228 6129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:33.599295 6129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:33.609234 6135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.609295 6134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.609365 6129 server_base.cc:1034] running on GCE node
W20260620 14:30:33.609295 6137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.609987 6129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:33.610653 6129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:33.611830 6129 hybrid_clock.cc:648] HybridClock initialized: now 1781965833611810 us; error 35 us; skew 500 ppm
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:33.614538 6129 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:33.615448 6129 webserver.cc:460] Webserver started at http://127.4.58.65:38631/ using document root <none> and password file <none>
I20260620 14:30:33.615994 6129 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:33.616060 6129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:33.616261 6129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:33.618378 6129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "59d00b99d25a7bc916d05dd6ebac7ed6"
server_key_iv: "2ee68a0d9a62eed9eec763866d9ec1c5"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.618932 6129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "59d00b99d25a7bc916d05dd6ebac7ed6"
server_key_iv: "2ee68a0d9a62eed9eec763866d9ec1c5"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.622936 6129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:33.625377 6144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.626585 6129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:33.626698 6129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "59d00b99d25a7bc916d05dd6ebac7ed6"
server_key_iv: "2ee68a0d9a62eed9eec763866d9ec1c5"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.626801 6129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:33.644308 6129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:33.647535 6129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:33.647713 6129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:33.649271 6129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:33.649324 6129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.649379 6129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:33.649407 6129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:33.659385 6129 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:46447
I20260620 14:30:33.659416 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:46447 every 8 connection(s)
I20260620 14:30:33.660305 6129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:33.661607 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6129
I20260620 14:30:33.661731 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:33.662046 4329 external_mini_cluster.cc:1428] Setting key 73fa21b3f87051e33cfa77fcc18654fc
I20260620 14:30:33.662653 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.650126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:33319 (local address 127.4.58.126:38205)
0620 14:30:33.650248 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.650252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.651333 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 14:30:33.652398 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.652402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.652467 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.652527 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.653938 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.654622 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.655744 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.655995 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.656086 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.658722 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.658725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.658727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.658754 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.660174 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.660876 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.660879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.660881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.660978 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.661367 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.661372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.661374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.661612 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.661754 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.662375 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.662499 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:30:33.663527 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.650695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.65:33319)
0620 14:30:33.651186 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.651231 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:33.652143 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:33.652971 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:33.652978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:33.653302 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:33.653794 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.653808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.654743 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.654747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.655621 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.655632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.656146 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.656150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.656301 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.656851 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:33.656867 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:33.658551 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:33.660358 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.660364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.660377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.660712 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.661078 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.661081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.661083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.661258 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.661805 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:33.661811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:33.662107 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 14:30:33.662306 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 14:30:33.662607 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260620 14:30:33.664850 6260 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38205
I20260620 14:30:33.665091 6260 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:33.665561 6260 heartbeater.cc:511] Master 127.4.58.126:38205 requested a full tablet report, sending...
I20260620 14:30:33.668056 6057 ts_manager.cc:194] Registered new tserver with Master: c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447)
I20260620 14:30:33.669294 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:33319
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:33.675274 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.667244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.65:57065)
0620 14:30:33.667541 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.667555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:33.667678 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:33.667945 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:33.667948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:33.668152 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:33.668288 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.668295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.669497 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.669501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.670387 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.670398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.671089 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.671093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.671188 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.671647 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:33.671661 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:33.672074 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:33.673781 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.673785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.673787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.673990 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.674271 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.674274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.674275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.674342 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.674747 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:33.674750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:33.674829 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:30:33.674917 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:30:33.675059 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260620 14:30:33.675321 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.667310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:57065 (local address 127.4.58.126:38205)
0620 14:30:33.667414 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.667418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.667616 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:30:33.667782 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.667786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.667836 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.667918 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.668405 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.669342 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.670524 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.670860 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.670935 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.672269 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.672273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.672275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.672297 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.673652 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.674082 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.674085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.674086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.674158 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.674421 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.674424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.674425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.674647 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.674740 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.675000 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.675111 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:33.691951 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:33.799974 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:33.800256 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:33.800311 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:33.804498 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:33.804566 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:33.804605 6269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:30:33.804628 6269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:33.804682 6269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:33.810037 6269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:33.811112 6269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:33.812110 6269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:33.821568 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.821583 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:33.821559 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:33.821748 6269 server_base.cc:1034] running on GCE node
I20260620 14:30:33.822237 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:33.822865 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:33.824013 6269 hybrid_clock.cc:648] HybridClock initialized: now 1781965833823984 us; error 41 us; skew 500 ppm
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:33.826558 6269 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:33.827445 6269 webserver.cc:460] Webserver started at http://127.4.58.66:44767/ using document root <none> and password file <none>
I20260620 14:30:33.827979 6269 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:33.828044 6269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:33.828223 6269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:33.830248 6269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "0711672969144a378fbc571906e0e304"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "72b30b631fcd7494d58ed5c5c3533bbb"
server_key_iv: "2220419923d2054ce31ee0a036716147"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.830816 6269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "0711672969144a378fbc571906e0e304"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "72b30b631fcd7494d58ed5c5c3533bbb"
server_key_iv: "2220419923d2054ce31ee0a036716147"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.834578 6269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:33.837100 6284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.838382 6269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:33.838506 6269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0711672969144a378fbc571906e0e304"
format_stamp: "Formatted at 2026-06-20 14:30:33 on dist-test-slave-gqzk"
server_key: "72b30b631fcd7494d58ed5c5c3533bbb"
server_key_iv: "2220419923d2054ce31ee0a036716147"
server_key_version: "encryptionkey@0"
I20260620 14:30:33.838614 6269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:33.863745 6269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:33.866928 6269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:33.867103 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:33.868637 6269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:33.868688 6269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:33.868741 6269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:33.868793 6269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:30:33 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:33.878338 6269 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:44861
I20260620 14:30:33.878400 6399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:44861 every 8 connection(s)
I20260620 14:30:33.879407 6269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:33.882071 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.869497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:54131 (local address 127.4.58.126:38205)
0620 14:30:33.869651 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.869656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.870511 (+ 855us) server_negotiation.cc:407] Connection header received
0620 14:30:33.871814 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.871818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.871883 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.871970 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.873503 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.874154 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.875401 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.875624 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.875714 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.878534 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.878538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.878541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.878569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.880098 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.880644 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.880648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.880650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.880730 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.881053 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.881056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.881058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.881236 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.881319 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.881791 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.881915 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:30:33.882833 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.869890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.66:54131)
0620 14:30:33.870346 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.870397 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:33.871523 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:33.872214 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:33.872224 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:33.872700 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:33.873299 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.873315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.874300 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.874303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.875280 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.875290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.875779 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.875783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.875953 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.876538 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:33.876561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:33.878307 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:33.880215 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.880221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.880236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.880554 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.880829 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.880833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.880835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.880967 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.881331 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:33.881338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:33.881573 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 14:30:33.881734 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 14:30:33.882049 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260620 14:30:33.884043 6400 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38205
I20260620 14:30:33.884281 6400 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:33.884873 6400 heartbeater.cc:511] Master 127.4.58.126:38205 requested a full tablet report, sending...
I20260620 14:30:33.886005 6056 ts_manager.cc:194] Registered new tserver with Master: 0711672969144a378fbc571906e0e304 (127.4.58.66:44861)
I20260620 14:30:33.886720 6056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:54131
I20260620 14:30:33.888257 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6269
I20260620 14:30:33.888374 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:33.888674 4329 external_mini_cluster.cc:1428] Setting key 5899214935e75ebeffa4ffefe9791191
I20260620 14:30:33.893795 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.886392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:46969 (local address 127.4.58.126:38205)
0620 14:30:33.886503 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:33.886506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:33.886667 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:30:33.886741 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:33.886743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:33.886781 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:33.886894 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:33.887340 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.888110 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.889171 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.889418 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.889497 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.890690 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:33.890693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:33.890695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:33.890715 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:33.891952 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.892407 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.892410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.892411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.892481 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:33.892719 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:33.892721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:33.892722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:33.893119 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:33.893286 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:33.893569 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:30:33.893681 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:33.893806 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:33.886289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.66:46969)
0620 14:30:33.886556 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:33.886570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:33.886660 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:33.886907 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:33.886909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:33.887109 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:33.887227 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.887234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.888229 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.888233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.889058 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:33.889068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:33.889523 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:33.889527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:33.889620 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:33.890152 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:33.890165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:33.890561 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:33.892052 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.892056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.892058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.892296 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.892555 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:33.892558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:33.892559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:33.892647 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:33.893280 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:33.893283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:33.893377 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:30:33.893479 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:30:33.893625 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:33.916190 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:34.029393 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:34.029663 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:34.029711 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:34.033828 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:34.033900 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:34.033942 6409 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:30:34.033955 6409 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:30:34.033986 6409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:34.039395 6409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:34.040457 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:34.041503 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:34.051120 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:34.051198 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:34.051120 6417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:34.051399 6409 server_base.cc:1034] running on GCE node
I20260620 14:30:34.051900 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:34.052548 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:34.053714 6409 hybrid_clock.cc:648] HybridClock initialized: now 1781965834053701 us; error 61 us; skew 500 ppm
Jun 20 14:30:34 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:34.056469 6409 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:34.057407 6409 webserver.cc:460] Webserver started at http://127.4.58.67:36235/ using document root <none> and password file <none>
I20260620 14:30:34.057963 6409 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:34.058030 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:34.058233 6409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:34.060297 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "9dcdb8ff9fe1400395929693cea76a01"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-gqzk"
server_key: "e217620780d10aa5a3d03285c146818c"
server_key_iv: "ec56308443648e8158fa4095815c046f"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.060907 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "9dcdb8ff9fe1400395929693cea76a01"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-gqzk"
server_key: "e217620780d10aa5a3d03285c146818c"
server_key_iv: "ec56308443648e8158fa4095815c046f"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.064527 6409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:34.067054 6424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:34.068341 6409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:34.068450 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "9dcdb8ff9fe1400395929693cea76a01"
format_stamp: "Formatted at 2026-06-20 14:30:34 on dist-test-slave-gqzk"
server_key: "e217620780d10aa5a3d03285c146818c"
server_key_iv: "ec56308443648e8158fa4095815c046f"
server_key_version: "encryptionkey@0"
I20260620 14:30:34.068544 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:34.091559 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:34.094681 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:34.094873 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:34.096283 6409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:34.096333 6409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:34.096386 6409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:34.096412 6409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:30:34 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:34.106386 6409 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:33393
I20260620 14:30:34.106503 6539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:33393 every 8 connection(s)
I20260620 14:30:34.107424 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:34.109679 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.097109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:58229 (local address 127.4.58.126:38205)
0620 14:30:34.097221 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.097225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.098329 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 14:30:34.099362 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.099366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.099427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.099478 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.101122 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.101714 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.103043 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.103369 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.103479 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.106222 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.106226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.106229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.106271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.107711 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.108229 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.108241 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.108243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.108321 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.108622 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.108625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.108627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.108853 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.108939 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.109398 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.109518 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:30:34.110432 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.097676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.67:58229)
0620 14:30:34.098167 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.098211 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.099105 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.099888 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.099898 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.100292 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.100974 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.100995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.101864 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.101867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.102928 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.102938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.103521 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.103526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.103726 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.104304 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.104334 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.106033 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.107818 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.107824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.107840 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.108121 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.108404 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.108407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.108409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.108537 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.108944 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.108950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.109178 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.109340 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.109647 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260620 14:30:34.111712 6540 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38205
I20260620 14:30:34.111843 6540 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:34.112334 6540 heartbeater.cc:511] Master 127.4.58.126:38205 requested a full tablet report, sending...
I20260620 14:30:34.112844 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6409
I20260620 14:30:34.112967 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:34.113266 4329 external_mini_cluster.cc:1428] Setting key c83d482daafb208f89fa18afeb6caba6
I20260620 14:30:34.113330 6057 ts_manager.cc:194] Registered new tserver with Master: 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:34.114037 6057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:58229
I20260620 14:30:34.115403 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:34.120926 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.113860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46047 (local address 127.4.58.126:38205)
0620 14:30:34.113968 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.113971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.114258 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:30:34.114426 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.114429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.114476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.114549 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.114986 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.115654 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.116515 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.116732 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.116846 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.118013 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.118016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.118019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.118045 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.119382 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.119806 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.119810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.119812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.119889 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.120130 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.120133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.120135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.120306 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.120391 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.120644 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.120778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:30:34.120939 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.113870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.4.58.67:46047)
0620 14:30:34.114186 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.114200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.114318 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.114587 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.114590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.114769 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.114879 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.114885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.115831 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.115834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.116403 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.116412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.116893 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.116897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.117003 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.117570 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.117581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.117882 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.119503 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.119507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.119509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.119718 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.119968 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.119971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.119973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.120051 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.120396 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.120399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.120488 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.120578 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.120724 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
Jun 20 14:30:34 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:30:34 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:34.140856 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.132806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.4.58.126:38205)
0620 14:30:34.132942 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.132946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.132958 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:34.132997 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.132999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.133038 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.133108 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.133798 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.134307 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.135142 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.135362 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.135449 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.137809 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.137812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.137814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.137842 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.139262 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.139647 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.139650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.139651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.139743 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.139996 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.139999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.140000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.140271 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.140393 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.140578 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.140678 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:30:34.143715 6057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:34.146826 6057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:34.161048 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.155058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.0.0.1:51504)
0620 14:30:34.155406 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.155452 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.155572 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.156470 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.156477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.156502 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.156640 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.156647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158153 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.158157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.159533 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.159543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.160493 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.160496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.160600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.160637 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.160744 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.160903 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260620 14:30:34.161059 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.155248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.4.58.65:46447)
0620 14:30:34.155944 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.155959 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.155983 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:30:34.156060 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.156065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.156281 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.156825 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.156977 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158036 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.159660 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.160379 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.160475 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.160646 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.160804 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.160902 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":93,"threads_started":1}
I20260620 14:30:34.161547 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.155257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:44861 (local address 127.0.0.1:44534)
0620 14:30:34.155608 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.155621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.155702 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.157464 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.157468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.157485 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.157563 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.157568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158761 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.158765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.159853 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.159860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.161130 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.161133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.161229 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.161252 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.161331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.161423 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260620 14:30:34.161689 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.156226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.4.58.66:44861)
0620 14:30:34.157007 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.157019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.157038 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:30:34.157098 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.157102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.157261 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.157480 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.157667 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158627 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.159960 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.160803 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.160904 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.161066 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.161460 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.161558 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":111,"threads_started":1}
I20260620 14:30:34.161548 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.155802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:33393 (local address 127.0.0.1:35984)
0620 14:30:34.156015 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.156028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.156107 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.157221 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.157224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.157240 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.157354 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.157360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158731 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.158734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.160008 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.160019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.161170 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.161173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.161256 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.161273 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.161341 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.161402 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260620 14:30:34.161852 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.156326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.4.58.67:33393)
0620 14:30:34.156773 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.156785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.156805 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:30:34.156882 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.156887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.157082 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.157229 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.157694 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.158586 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.160130 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.160901 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.160986 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.161121 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.161616 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.161716 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":124,"threads_started":1}
I20260620 14:30:34.164685 6194 tablet_service.cc:1467] Processing CreateTablet for tablet 20960de6baeb41968a2375c55e6df267 (DEFAULT_TABLE table=test-table [id=a31525f6de3d4d719876d6bc443c138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.164701 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 20960de6baeb41968a2375c55e6df267 (DEFAULT_TABLE table=test-table [id=a31525f6de3d4d719876d6bc443c138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.165889 6194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20960de6baeb41968a2375c55e6df267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.165889 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20960de6baeb41968a2375c55e6df267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.167366 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 20960de6baeb41968a2375c55e6df267 (DEFAULT_TABLE table=test-table [id=a31525f6de3d4d719876d6bc443c138d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.168345 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20960de6baeb41968a2375c55e6df267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.174278 6563 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Bootstrap starting.
I20260620 14:30:34.174772 6565 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Bootstrap starting.
I20260620 14:30:34.174825 6564 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Bootstrap starting.
I20260620 14:30:34.176775 6564 tablet_bootstrap.cc:654] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.176926 6563 tablet_bootstrap.cc:654] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.176926 6565 tablet_bootstrap.cc:654] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.177553 6564 log.cc:824] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.177745 6563 log.cc:824] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.177745 6565 log.cc:824] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:34.179706 6565 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: No bootstrap required, opened a new log
I20260620 14:30:34.179706 6564 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: No bootstrap required, opened a new log
I20260620 14:30:34.179893 6564 ts_tablet_manager.cc:1397] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:30:34.179895 6565 ts_tablet_manager.cc:1397] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:30:34.182782 6565 raft_consensus.cc:348] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.182783 6564 raft_consensus.cc:348] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.183007 6565 raft_consensus.cc:374] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.183008 6564 raft_consensus.cc:374] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.183048 6565 raft_consensus.cc:729] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcdb8ff9fe1400395929693cea76a01, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.183061 6564 raft_consensus.cc:729] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711672969144a378fbc571906e0e304, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.183481 6565 consensus_queue.cc:260] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.183491 6564 consensus_queue.cc:260] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.184319 6540 heartbeater.cc:503] Master 127.4.58.126:38205 was elected leader, sending a full tablet report...
I20260620 14:30:34.184470 6565 ts_tablet_manager.cc:1428] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:30:34.184475 6564 ts_tablet_manager.cc:1428] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:30:34.184692 6563 tablet_bootstrap.cc:492] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: No bootstrap required, opened a new log
I20260620 14:30:34.184903 6563 ts_tablet_manager.cc:1397] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:30:34.185052 6400 heartbeater.cc:503] Master 127.4.58.126:38205 was elected leader, sending a full tablet report...
I20260620 14:30:34.188278 6563 raft_consensus.cc:348] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.188541 6563 raft_consensus.cc:374] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.188593 6563 raft_consensus.cc:729] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.189054 6563 consensus_queue.cc:260] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.189615 6260 heartbeater.cc:503] Master 127.4.58.126:38205 was elected leader, sending a full tablet report...
I20260620 14:30:34.189863 6563 ts_tablet_manager.cc:1428] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 14:30:34.358649 6541 tablet.cc:2369] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:34.380661 6401 tablet.cc:2369] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:34.391690 6571 raft_consensus.cc:482] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:34.391924 6571 raft_consensus.cc:504] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.393191 6571 leader_election.cc:288] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393), 0711672969144a378fbc571906e0e304 (127.4.58.66:44861)
I20260620 14:30:34.397207 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.393513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:33393 (local address 127.4.58.65:58573)
0620 14:30:34.393891 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.393903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.393981 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.394215 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.394218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.394235 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.394370 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.394377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.395122 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.395126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.396038 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.396047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.396660 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.396859 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.396884 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.396965 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.397037 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":76,"thread_start_us":100,"threads_started":1}
I20260620 14:30:34.397214 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.393623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:58657 (local address 127.4.58.66:44861)
0620 14:30:34.393874 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.393878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.393891 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:34.394143 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.394146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.394193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.394261 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.394525 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.395038 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396157 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.396547 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.396703 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.397022 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.397101 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260620 14:30:34.397199 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.393607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:58573 (local address 127.4.58.67:33393)
0620 14:30:34.393795 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.393800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.393971 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:30:34.394061 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.394064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.394117 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.394203 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.394478 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.395011 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396161 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.396550 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396644 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.396715 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.396994 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.397068 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 14:30:34.397555 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.393496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:44861 (local address 127.4.58.65:58657)
0620 14:30:34.393808 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.393835 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.393961 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.394297 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.394301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.394320 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.394419 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.394424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.395145 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.395148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.396035 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.396045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.396659 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.396663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.396856 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.396880 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.397351 (+ 471us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.397426 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":68,"thread_start_us":137,"threads_started":1}
I20260620 14:30:34.397907 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20960de6baeb41968a2375c55e6df267" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01" is_pre_election: true
I20260620 14:30:34.398108 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20960de6baeb41968a2375c55e6df267" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304" is_pre_election: true
I20260620 14:30:34.398192 6494 raft_consensus.cc:2454] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:34.398392 6354 raft_consensus.cc:2454] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:34.398680 6147 leader_election.cc:302] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:34.398974 6571 raft_consensus.cc:2781] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:34.399066 6571 raft_consensus.cc:482] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:34.399116 6571 raft_consensus.cc:3037] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.400110 6571 raft_consensus.cc:504] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.400434 6571 leader_election.cc:288] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Requested vote from peers 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393), 0711672969144a378fbc571906e0e304 (127.4.58.66:44861)
I20260620 14:30:34.400741 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20960de6baeb41968a2375c55e6df267" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01"
I20260620 14:30:34.400772 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20960de6baeb41968a2375c55e6df267" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304"
I20260620 14:30:34.400892 6494 raft_consensus.cc:3037] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.400904 6354 raft_consensus.cc:3037] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.401867 6354 raft_consensus.cc:2454] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:34.402153 6146 leader_election.cc:302] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:34.402344 6571 raft_consensus.cc:2781] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:34.402484 6494 raft_consensus.cc:2454] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:34.402542 6571 raft_consensus.cc:686] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Becoming Leader. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Running, Role: LEADER
I20260620 14:30:34.402935 6571 consensus_queue.cc:237] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } }
I20260620 14:30:34.405963 6056 catalog_manager.cc:5495] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65). New cstate: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } health_report { overall_health: UNKNOWN } } }
W20260620 14:30:34.411595 6261 tablet.cc:2369] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:30:34 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:34.487331 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.478782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.4.58.126:38205)
0620 14:30:34.478923 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.478929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.479051 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:30:34.479192 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.479195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.479231 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.479304 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:34.480018 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.480557 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.481373 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.481573 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.481658 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.483996 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:34.484000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:34.484002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:34.484026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:34.485464 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.485967 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.485970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.485971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.486052 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:34.486434 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:34.486437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:34.486438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:34.486623 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:34.486702 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.487054 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.487193 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":53}
I20260620 14:30:34.487332 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.478587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38205 (local address 127.0.0.1:47974)
0620 14:30:34.478959 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.478974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.479101 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.479331 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.479333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.479771 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:34.479896 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.479903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.480663 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.480667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.481266 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.481275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.481669 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.481672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.481784 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.482338 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:34.482351 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:34.483859 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:34.485581 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.485590 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.485596 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.485861 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.486233 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:34.486235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:34.486237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:34.486326 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:34.486744 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:34.486756 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:34.486883 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.486992 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.487131 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260620 14:30:34.491528 6056 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:47974:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:30:34.492226 6056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:34.496860 6194 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8c10358c004cab8b8cda0330307e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6510692d589d462abc0bab84b18f7d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:34.496986 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8c10358c004cab8b8cda0330307e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6510692d589d462abc0bab84b18f7d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:34.496986 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8c10358c004cab8b8cda0330307e96 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6510692d589d462abc0bab84b18f7d87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:30:34.497664 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8c10358c004cab8b8cda0330307e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.497664 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8c10358c004cab8b8cda0330307e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.497666 6194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8c10358c004cab8b8cda0330307e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.500290 6563 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: Bootstrap starting.
I20260620 14:30:34.500559 6564 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: Bootstrap starting.
I20260620 14:30:34.500912 6565 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: Bootstrap starting.
I20260620 14:30:34.501608 6563 tablet_bootstrap.cc:654] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.501631 6564 tablet_bootstrap.cc:654] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.501896 6565 tablet_bootstrap.cc:654] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.503023 6564 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: No bootstrap required, opened a new log
I20260620 14:30:34.503111 6564 ts_tablet_manager.cc:1397] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:34.503183 6565 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: No bootstrap required, opened a new log
I20260620 14:30:34.503257 6565 ts_tablet_manager.cc:1397] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:34.503556 6564 raft_consensus.cc:348] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.503638 6565 raft_consensus.cc:348] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.503690 6564 raft_consensus.cc:374] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.503729 6564 raft_consensus.cc:729] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711672969144a378fbc571906e0e304, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.503734 6565 raft_consensus.cc:374] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.503769 6565 raft_consensus.cc:729] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcdb8ff9fe1400395929693cea76a01, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.503865 6564 consensus_queue.cc:260] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.503881 6565 consensus_queue.cc:260] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.504124 6565 ts_tablet_manager.cc:1428] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:30:34.504124 6564 ts_tablet_manager.cc:1428] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:30:34.504129 6569 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.504333 6569 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.504258 6570 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.504384 6570 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.504702 6563 tablet_bootstrap.cc:492] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: No bootstrap required, opened a new log
I20260620 14:30:34.504818 6563 ts_tablet_manager.cc:1397] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 14:30:34.505285 6563 raft_consensus.cc:348] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.505385 6563 raft_consensus.cc:374] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.505415 6563 raft_consensus.cc:729] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.505537 6563 consensus_queue.cc:260] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.505784 6563 ts_tablet_manager.cc:1428] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:30:34.505781 6571 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.505949 6571 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.617822 6571 raft_consensus.cc:482] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:34.617969 6571 raft_consensus.cc:504] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.618424 6571 leader_election.cc:288] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:34.618884 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8c10358c004cab8b8cda0330307e96" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01" is_pre_election: true
I20260620 14:30:34.618896 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8c10358c004cab8b8cda0330307e96" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304" is_pre_election: true
I20260620 14:30:34.619036 6354 raft_consensus.cc:2454] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:34.619040 6494 raft_consensus.cc:2454] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:34.619321 6146 leader_election.cc:302] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:34.619506 6571 raft_consensus.cc:2781] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:34.619562 6571 raft_consensus.cc:482] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:34.619593 6571 raft_consensus.cc:3037] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.620563 6571 raft_consensus.cc:504] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.620905 6571 leader_election.cc:288] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Requested vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:34.621274 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8c10358c004cab8b8cda0330307e96" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01"
I20260620 14:30:34.621272 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8c10358c004cab8b8cda0330307e96" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304"
I20260620 14:30:34.621384 6494 raft_consensus.cc:3037] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.621395 6354 raft_consensus.cc:3037] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.622438 6494 raft_consensus.cc:2454] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:34.622465 6354 raft_consensus.cc:2454] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:34.622691 6147 leader_election.cc:302] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:34.622855 6571 raft_consensus.cc:2781] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:34.622941 6571 raft_consensus.cc:686] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Becoming Leader. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Running, Role: LEADER
I20260620 14:30:34.623073 6571 consensus_queue.cc:237] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.623711 6571 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: TxnStatusTablet state changed. Reason: New leader c7c5f7ed4ca44f25b24ee6410bca29ed. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.623798 6571 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:34.624060 6586 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:30:34.624307 6586 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:30:34.624804 6057 catalog_manager.cc:5495] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65). New cstate: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:34.663403 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.658706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.4.58.65:46447)
0620 14:30:34.658839 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.658843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.658876 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:30:34.659063 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.659067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.659147 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.659258 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.659632 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.660366 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.661591 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.661922 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.662019 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.662443 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.662982 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.663096 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.663197 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.663262 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:30:34.663424 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.658566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.0.0.1:51518)
0620 14:30:34.658819 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.658835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.658933 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.659386 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.659390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.659398 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:34.659504 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.659510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.660479 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.660483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.661467 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.661478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.662111 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.662115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.662209 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.662261 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.663088 (+ 827us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.663106 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.663179 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.663278 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 14:30:34.669459 6494 raft_consensus.cc:1261] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.670205 6574 consensus_queue.cc:1035] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.670593 6354 raft_consensus.cc:1261] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.671166 6574 consensus_queue.cc:1035] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.674762 6570 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: TxnStatusTablet state changed. Reason: New leader c7c5f7ed4ca44f25b24ee6410bca29ed. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.674885 6570 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.675949 6569 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: TxnStatusTablet state changed. Reason: New leader c7c5f7ed4ca44f25b24ee6410bca29ed. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.676071 6569 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.678982 6574 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.679113 6574 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:34.679229 6574 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.679301 6574 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P c7c5f7ed4ca44f25b24ee6410bca29ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:30:34.679330 6570 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.679454 6570 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 9dcdb8ff9fe1400395929693cea76a01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.680703 6569 tablet_replica.cc:439] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } } }
I20260620 14:30:34.680846 6569 tablet_replica.cc:442] T 1b8c10358c004cab8b8cda0330307e96 P 0711672969144a378fbc571906e0e304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:30:34.683229 6590 mvcc.cc:205] Tried to move back new op lower bound from 7298932058798735360 to 7298932058616467456. Current Snapshot: MvccSnapshot[applied={T|T < 7298932058798735360}]
I20260620 14:30:34.688489 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.685181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.4.58.65:46447)
0620 14:30:34.685300 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.685303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.685315 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:34.685464 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.685466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.685516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.685567 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.685860 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.686371 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.687575 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.687785 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.687865 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.688104 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.688170 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.688216 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.688340 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.688384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:30:34.698755 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.695583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:37863 (local address 127.4.58.65:46447)
0620 14:30:34.695686 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.695691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.695885 (+ 194us) server_negotiation.cc:407] Connection header received
0620 14:30:34.695992 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.695995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.696034 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.696090 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.696374 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.696900 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.697725 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.697937 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.698016 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.698300 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.698375 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.698430 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.698510 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.698586 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":32}
I20260620 14:30:34.698763 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.695536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.4.58.65:37863)
0620 14:30:34.695806 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.695818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.695920 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.696102 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.696105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.696112 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:34.696247 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.696253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.697002 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.697005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.697625 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.697633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.698023 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.698026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.698127 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.698169 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.698442 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.698448 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.698503 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.698599 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260620 14:30:34.705248 6354 raft_consensus.cc:1261] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.705292 6494 raft_consensus.cc:1261] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.705655 6574 consensus_queue.cc:1035] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.705811 6571 consensus_queue.cc:1035] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.709396 6589 mvcc.cc:205] Tried to move back new op lower bound from 7298932058949517312 to 7298932057716183040. Current Snapshot: MvccSnapshot[applied={T|T < 7298932058949517312}]
I20260620 14:30:34.709962 6592 mvcc.cc:205] Tried to move back new op lower bound from 7298932058949517312 to 7298932057716183040. Current Snapshot: MvccSnapshot[applied={T|T < 7298932058949517312}]
I20260620 14:30:34.746994 6056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:34.747195 6056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:34.749699 6056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Sending DeleteTablet for 3 replicas of tablet 20960de6baeb41968a2375c55e6df267
I20260620 14:30:34.750391 6193 tablet_service.cc:1514] Processing DeleteTablet for tablet 20960de6baeb41968a2375c55e6df267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:34 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51504
I20260620 14:30:34.750530 6474 tablet_service.cc:1514] Processing DeleteTablet for tablet 20960de6baeb41968a2375c55e6df267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:34 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35984
I20260620 14:30:34.750689 6334 tablet_service.cc:1514] Processing DeleteTablet for tablet 20960de6baeb41968a2375c55e6df267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:34 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44534
I20260620 14:30:34.750952 6606 tablet_replica.cc:330] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: stopping tablet replica
I20260620 14:30:34.751106 6607 tablet_replica.cc:330] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: stopping tablet replica
I20260620 14:30:34.751278 6606 raft_consensus.cc:2229] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:34.751457 6607 raft_consensus.cc:2229] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:34.751695 6607 raft_consensus.cc:2258] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:34.751696 6606 raft_consensus.cc:2258] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:34.751933 6056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:30:34.752106 6608 tablet_replica.cc:330] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: stopping tablet replica
W20260620 14:30:34.752399 6056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:34.752689 6608 raft_consensus.cc:2229] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:34.752976 6608 raft_consensus.cc:2258] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:34.753229 6607 ts_tablet_manager.cc:1905] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:34.754254 6608 ts_tablet_manager.cc:1905] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:34.756186 6607 ts_tablet_manager.cc:1918] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:34.756285 6607 log.cc:1196] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/20960de6baeb41968a2375c55e6df267
I20260620 14:30:34.756587 6607 ts_tablet_manager.cc:1939] T 20960de6baeb41968a2375c55e6df267 P 9dcdb8ff9fe1400395929693cea76a01: Deleting consensus metadata
I20260620 14:30:34.757385 6608 ts_tablet_manager.cc:1918] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:34.757931 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 898a38d56b194f8d944e31c8abcb568f (DEFAULT_TABLE table=test-table [id=b81716c509504cda934e4d949701daaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.757959 6193 tablet_service.cc:1467] Processing CreateTablet for tablet 898a38d56b194f8d944e31c8abcb568f (DEFAULT_TABLE table=test-table [id=b81716c509504cda934e4d949701daaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.758157 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898a38d56b194f8d944e31c8abcb568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.758232 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898a38d56b194f8d944e31c8abcb568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.759745 6565 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Bootstrap starting.
I20260620 14:30:34.759819 6043 catalog_manager.cc:4841] TS 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393): tablet 20960de6baeb41968a2375c55e6df267 (table test-table [id=a31525f6de3d4d719876d6bc443c138d]) successfully deleted
I20260620 14:30:34.760460 6608 log.cc:1196] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/20960de6baeb41968a2375c55e6df267
I20260620 14:30:34.760609 6565 tablet_bootstrap.cc:654] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.760779 6563 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Bootstrap starting.
I20260620 14:30:34.760833 6608 ts_tablet_manager.cc:1939] T 20960de6baeb41968a2375c55e6df267 P 0711672969144a378fbc571906e0e304: Deleting consensus metadata
I20260620 14:30:34.761310 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 898a38d56b194f8d944e31c8abcb568f (DEFAULT_TABLE table=test-table [id=b81716c509504cda934e4d949701daaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:34.761555 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898a38d56b194f8d944e31c8abcb568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:34.761634 6563 tablet_bootstrap.cc:654] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.762066 6042 catalog_manager.cc:4841] TS 0711672969144a378fbc571906e0e304 (127.4.58.66:44861): tablet 20960de6baeb41968a2375c55e6df267 (table test-table [id=a31525f6de3d4d719876d6bc443c138d]) successfully deleted
I20260620 14:30:34.762552 6565 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: No bootstrap required, opened a new log
I20260620 14:30:34.762630 6565 ts_tablet_manager.cc:1397] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:30:34.763000 6565 raft_consensus.cc:348] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.763095 6565 raft_consensus.cc:374] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.763123 6565 raft_consensus.cc:729] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcdb8ff9fe1400395929693cea76a01, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.763240 6565 consensus_queue.cc:260] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.763465 6565 ts_tablet_manager.cc:1428] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:34.763960 6564 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Bootstrap starting.
I20260620 14:30:34.764329 6563 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: No bootstrap required, opened a new log
I20260620 14:30:34.764420 6563 ts_tablet_manager.cc:1397] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:30:34.764909 6564 tablet_bootstrap.cc:654] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:34.764871 6563 raft_consensus.cc:348] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.764976 6563 raft_consensus.cc:374] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.765015 6563 raft_consensus.cc:729] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.765147 6563 consensus_queue.cc:260] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.765415 6563 ts_tablet_manager.cc:1428] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:34.766196 6564 tablet_bootstrap.cc:492] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: No bootstrap required, opened a new log
I20260620 14:30:34.766275 6564 ts_tablet_manager.cc:1397] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:34.766669 6564 raft_consensus.cc:348] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.766772 6564 raft_consensus.cc:374] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:34.766803 6564 raft_consensus.cc:729] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711672969144a378fbc571906e0e304, State: Initialized, Role: FOLLOWER
I20260620 14:30:34.766899 6606 ts_tablet_manager.cc:1905] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:34.766925 6564 consensus_queue.cc:260] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.767169 6564 ts_tablet_manager.cc:1428] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:34.769971 6606 ts_tablet_manager.cc:1918] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:34.770077 6606 log.cc:1196] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/20960de6baeb41968a2375c55e6df267
I20260620 14:30:34.770380 6606 ts_tablet_manager.cc:1939] T 20960de6baeb41968a2375c55e6df267 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting consensus metadata
I20260620 14:30:34.771554 6043 catalog_manager.cc:4841] TS c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447): tablet 20960de6baeb41968a2375c55e6df267 (table test-table [id=a31525f6de3d4d719876d6bc443c138d]) successfully deleted
I20260620 14:30:34.847537 6569 raft_consensus.cc:482] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:34.847692 6569 raft_consensus.cc:504] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.849030 6569 leader_election.cc:288] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:34.853713 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.849812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:57921 (local address 127.4.58.67:33393)
0620 14:30:34.849920 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.849924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.850120 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:30:34.850230 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.850233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.850280 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.850373 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.850591 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.851293 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.852692 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.853144 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.853244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.853311 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.853525 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.853596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:30:34.853742 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.849677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:33393 (local address 127.4.58.66:57921)
0620 14:30:34.850041 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.850054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.850151 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.850380 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.850383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.850399 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.850496 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.850501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.851416 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.851420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.852550 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.852557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.853275 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.853279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.853403 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.853426 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.853493 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.853567 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":25,"spinlock_wait_cycles":384,"thread_start_us":99,"threads_started":1}
I20260620 14:30:34.853945 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.849338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:60653 (local address 127.4.58.65:46447)
0620 14:30:34.849480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.849485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.849628 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:30:34.849823 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.849826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.849879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.849931 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:34.850357 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.850933 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.852509 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.853185 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.853288 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.853355 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.853733 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.853823 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:30:34.854293 6612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.849230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.4.58.66:60653)
0620 14:30:34.849544 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.849558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.849669 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.850074 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.850077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.850096 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:34.850243 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.850250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.851035 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.851039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.852340 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.852351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.853436 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.853440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.853591 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.853615 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.854081 (+ 466us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.854156 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":38,"spinlock_wait_cycles":19200,"thread_start_us":161,"threads_started":1}
I20260620 14:30:34.854346 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898a38d56b194f8d944e31c8abcb568f" candidate_uuid: "0711672969144a378fbc571906e0e304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01" is_pre_election: true
I20260620 14:30:34.854501 6494 raft_consensus.cc:2454] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711672969144a378fbc571906e0e304 in term 0.
I20260620 14:30:34.854876 6287 leader_election.cc:302] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, 9dcdb8ff9fe1400395929693cea76a01; no voters:
I20260620 14:30:34.855083 6214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898a38d56b194f8d944e31c8abcb568f" candidate_uuid: "0711672969144a378fbc571906e0e304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" is_pre_election: true
I20260620 14:30:34.855158 6569 raft_consensus.cc:2781] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:34.855214 6569 raft_consensus.cc:482] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:34.855257 6569 raft_consensus.cc:3037] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.855327 6214 raft_consensus.cc:2454] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0711672969144a378fbc571906e0e304 in term 0.
I20260620 14:30:34.856438 6569 raft_consensus.cc:504] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.856843 6569 leader_election.cc:288] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [CANDIDATE]: Term 1 election: Requested vote from peers c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:34.857227 6214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898a38d56b194f8d944e31c8abcb568f" candidate_uuid: "0711672969144a378fbc571906e0e304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed"
I20260620 14:30:34.857337 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898a38d56b194f8d944e31c8abcb568f" candidate_uuid: "0711672969144a378fbc571906e0e304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01"
I20260620 14:30:34.857394 6214 raft_consensus.cc:3037] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.857443 6494 raft_consensus.cc:3037] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:34.858492 6494 raft_consensus.cc:2454] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711672969144a378fbc571906e0e304 in term 1.
I20260620 14:30:34.858515 6214 raft_consensus.cc:2454] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0711672969144a378fbc571906e0e304 in term 1.
I20260620 14:30:34.858836 6287 leader_election.cc:302] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, 9dcdb8ff9fe1400395929693cea76a01; no voters:
I20260620 14:30:34.859002 6569 raft_consensus.cc:2781] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:34.859249 6569 raft_consensus.cc:686] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 1 LEADER]: Becoming Leader. State: Replica: 0711672969144a378fbc571906e0e304, State: Running, Role: LEADER
I20260620 14:30:34.859742 6569 consensus_queue.cc:237] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:34.862177 6056 catalog_manager.cc:5495] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0711672969144a378fbc571906e0e304 (127.4.58.66). New cstate: current_term: 1 leader_uuid: "0711672969144a378fbc571906e0e304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:34.880772 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.876682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.4.58.66:44861)
0620 14:30:34.876840 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.876845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.876894 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:30:34.877073 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.877075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.877127 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.877217 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.877460 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.877976 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.879104 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.879484 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.879589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.879854 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.880373 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.880478 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.880557 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.880618 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:34.899744 6559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.892714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:57545 (local address 127.4.58.65:46447)
0620 14:30:34.892879 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.892883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.896780 (+ 3897us) server_negotiation.cc:407] Connection header received
0620 14:30:34.896949 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.896952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.897005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.897067 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.897311 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.897903 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.898742 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.898965 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.899051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.899331 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.899402 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.899459 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.899545 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.899597 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:30:34.899744 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.892632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.4.58.66:57545)
0620 14:30:34.896630 (+ 3998us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.896645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.896816 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.897091 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.897094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.897101 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:34.897206 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.897211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.898006 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.898010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.898630 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.898638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.899065 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.899068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.899152 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.899201 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.899471 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.899477 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.899535 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.899612 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3892,"thread_start_us":147,"threads_started":1}
I20260620 14:30:34.906701 6494 raft_consensus.cc:1261] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Refusing update from remote peer 0711672969144a378fbc571906e0e304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.906986 6214 raft_consensus.cc:1261] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Refusing update from remote peer 0711672969144a378fbc571906e0e304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:34.907557 6614 consensus_queue.cc:1035] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.907729 6569 consensus_queue.cc:1035] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:34.926062 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.922669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:44861 (local address 127.4.58.65:54929)
0620 14:30:34.922954 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:34.922969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:34.923043 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:34.923377 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:34.923380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:34.923389 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:30:34.923506 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.923512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.924330 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.924333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.925018 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:34.925028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.925434 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.925438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:34.925517 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.925547 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.925800 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.925807 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:30:34.925865 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:30:34.925952 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260620 14:30:34.926055 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:34.922827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:54929 (local address 127.4.58.66:44861)
0620 14:30:34.923177 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:34.923181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:34.923193 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:34.923229 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:34.923231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:34.923278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:34.923345 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:34.923603 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.924230 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.925120 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:34.925344 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:34.925428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:34.925651 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:34.925724 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:34.925782 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:34.925875 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:30:34.925921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
W20260620 14:30:34.926884 6592 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:30:36.486202 6054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:36.486368 6054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:36.488729 6054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Sending DeleteTablet for 3 replicas of tablet 898a38d56b194f8d944e31c8abcb568f
I20260620 14:30:36.489303 6334 tablet_service.cc:1514] Processing DeleteTablet for tablet 898a38d56b194f8d944e31c8abcb568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44534
I20260620 14:30:36.489358 6193 tablet_service.cc:1514] Processing DeleteTablet for tablet 898a38d56b194f8d944e31c8abcb568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51504
I20260620 14:30:36.489398 6474 tablet_service.cc:1514] Processing DeleteTablet for tablet 898a38d56b194f8d944e31c8abcb568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35984
I20260620 14:30:36.489656 6643 tablet_replica.cc:330] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: stopping tablet replica
I20260620 14:30:36.489727 6645 tablet_replica.cc:330] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: stopping tablet replica
I20260620 14:30:36.489781 6643 raft_consensus.cc:2229] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:36.489878 6645 raft_consensus.cc:2229] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:36.489995 6645 raft_consensus.cc:2258] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.490124 6643 raft_consensus.cc:2258] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.490435 6644 tablet_replica.cc:330] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: stopping tablet replica
I20260620 14:30:36.490585 6644 raft_consensus.cc:2229] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:36.490620 6645 ts_tablet_manager.cc:1905] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:36.490738 6644 raft_consensus.cc:2258] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.490911 6643 ts_tablet_manager.cc:1905] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:36.491418 6644 ts_tablet_manager.cc:1905] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:36.493306 6645 ts_tablet_manager.cc:1918] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:30:36.493391 6645 log.cc:1196] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/898a38d56b194f8d944e31c8abcb568f
I20260620 14:30:36.493530 6643 ts_tablet_manager.cc:1918] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:30:36.493606 6643 log.cc:1196] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/898a38d56b194f8d944e31c8abcb568f
I20260620 14:30:36.493629 6645 ts_tablet_manager.cc:1939] T 898a38d56b194f8d944e31c8abcb568f P 9dcdb8ff9fe1400395929693cea76a01: Deleting consensus metadata
I20260620 14:30:36.493932 6643 ts_tablet_manager.cc:1939] T 898a38d56b194f8d944e31c8abcb568f P 0711672969144a378fbc571906e0e304: Deleting consensus metadata
I20260620 14:30:36.493995 6644 ts_tablet_manager.cc:1918] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:30:36.494064 6644 log.cc:1196] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/898a38d56b194f8d944e31c8abcb568f
I20260620 14:30:36.494297 6644 ts_tablet_manager.cc:1939] T 898a38d56b194f8d944e31c8abcb568f P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting consensus metadata
I20260620 14:30:36.494400 6043 catalog_manager.cc:4841] TS 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393): tablet 898a38d56b194f8d944e31c8abcb568f (table test-table [id=b81716c509504cda934e4d949701daaf]) successfully deleted
I20260620 14:30:36.494617 6042 catalog_manager.cc:4841] TS 0711672969144a378fbc571906e0e304 (127.4.58.66:44861): tablet 898a38d56b194f8d944e31c8abcb568f (table test-table [id=b81716c509504cda934e4d949701daaf]) successfully deleted
I20260620 14:30:36.495004 6043 catalog_manager.cc:4841] TS c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447): tablet 898a38d56b194f8d944e31c8abcb568f (table test-table [id=b81716c509504cda934e4d949701daaf]) successfully deleted
Jun 20 14:30:36 dist-test-slave-gqzk krb5kdc[6012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:30:36 dist-test-slave-gqzk krb5kdc[6012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965836, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:36.521282 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.509546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.4.58.126:38205)
0620 14:30:36.509847 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.509856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.509872 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:30:36.509983 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.509987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.510031 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.510106 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:36.510869 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.511491 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.512402 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.512655 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.512746 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.518101 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:36.518105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:36.518107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:36.518138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:36.519572 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.520025 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.520028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.520030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.520116 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:36.520367 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:36.520371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:36.520373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:36.520569 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:36.520693 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.520973 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.521116 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260620 14:30:36.523927 6054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:36.524384 6054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:36.529204 6193 tablet_service.cc:1467] Processing CreateTablet for tablet 3905631bc00442a38b40acacc5121556 (DEFAULT_TABLE table=test-table [id=eaa4759c180742c89b03bb54a6957927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.529238 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 3905631bc00442a38b40acacc5121556 (DEFAULT_TABLE table=test-table [id=eaa4759c180742c89b03bb54a6957927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.529546 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905631bc00442a38b40acacc5121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.529549 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905631bc00442a38b40acacc5121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.531692 6654 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Bootstrap starting.
I20260620 14:30:36.531915 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 3905631bc00442a38b40acacc5121556 (DEFAULT_TABLE table=test-table [id=eaa4759c180742c89b03bb54a6957927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.532191 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905631bc00442a38b40acacc5121556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.532701 6654 tablet_bootstrap.cc:654] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.533499 6655 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Bootstrap starting.
I20260620 14:30:36.533903 6654 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: No bootstrap required, opened a new log
I20260620 14:30:36.533989 6654 ts_tablet_manager.cc:1397] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:36.534404 6655 tablet_bootstrap.cc:654] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.534379 6654 raft_consensus.cc:348] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.534489 6654 raft_consensus.cc:374] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.534523 6654 raft_consensus.cc:729] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711672969144a378fbc571906e0e304, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.534641 6654 consensus_queue.cc:260] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.534897 6654 ts_tablet_manager.cc:1428] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:36.534965 6657 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Bootstrap starting.
I20260620 14:30:36.535485 6655 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: No bootstrap required, opened a new log
I20260620 14:30:36.535569 6655 ts_tablet_manager.cc:1397] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:36.535846 6657 tablet_bootstrap.cc:654] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.535964 6655 raft_consensus.cc:348] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.536084 6655 raft_consensus.cc:374] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.536119 6655 raft_consensus.cc:729] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.536259 6655 consensus_queue.cc:260] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.536525 6655 ts_tablet_manager.cc:1428] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:36.537451 6657 tablet_bootstrap.cc:492] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: No bootstrap required, opened a new log
I20260620 14:30:36.537523 6657 ts_tablet_manager.cc:1397] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:36.537892 6657 raft_consensus.cc:348] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.537995 6657 raft_consensus.cc:374] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.538031 6657 raft_consensus.cc:729] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcdb8ff9fe1400395929693cea76a01, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.538151 6657 consensus_queue.cc:260] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.538512 6657 ts_tablet_manager.cc:1428] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:36.616654 6571 raft_consensus.cc:482] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:36.616829 6571 raft_consensus.cc:504] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.617288 6571 leader_election.cc:288] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:36.617722 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905631bc00442a38b40acacc5121556" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304" is_pre_election: true
I20260620 14:30:36.617719 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905631bc00442a38b40acacc5121556" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01" is_pre_election: true
I20260620 14:30:36.617871 6494 raft_consensus.cc:2454] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:36.617877 6354 raft_consensus.cc:2454] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:36.618207 6147 leader_election.cc:302] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:36.618422 6571 raft_consensus.cc:2781] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:36.618489 6571 raft_consensus.cc:482] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:36.618515 6571 raft_consensus.cc:3037] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.619469 6571 raft_consensus.cc:504] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.619786 6571 leader_election.cc:288] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Requested vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:36.620124 6354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905631bc00442a38b40acacc5121556" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304"
I20260620 14:30:36.620136 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905631bc00442a38b40acacc5121556" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01"
I20260620 14:30:36.620240 6354 raft_consensus.cc:3037] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.620270 6494 raft_consensus.cc:3037] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.621192 6354 raft_consensus.cc:2454] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:36.621225 6494 raft_consensus.cc:2454] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:36.621482 6146 leader_election.cc:302] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0711672969144a378fbc571906e0e304, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:36.621627 6571 raft_consensus.cc:2781] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:36.621712 6571 raft_consensus.cc:686] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Becoming Leader. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Running, Role: LEADER
I20260620 14:30:36.621852 6571 consensus_queue.cc:237] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.623538 6054 catalog_manager.cc:5495] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65). New cstate: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:36.641040 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:36.637868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.4.58.65:46447)
0620 14:30:36.638128 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:36.638155 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:36.638165 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:36.638210 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:36.638211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:36.638251 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:36.638319 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:36.638578 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.639113 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.639999 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:36.640237 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:36.640325 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:36.640575 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:36.640648 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:36.640705 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:36.640839 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:30:36.640894 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260620 14:30:36.647154 6494 raft_consensus.cc:1261] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:36.647195 6354 raft_consensus.cc:1261] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:36.647570 6571 consensus_queue.cc:1035] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:36.647714 6630 consensus_queue.cc:1035] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:36.681173 6054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:36.681335 6054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:36.683521 6054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Sending DeleteTablet for 3 replicas of tablet 3905631bc00442a38b40acacc5121556
I20260620 14:30:36.684084 6193 tablet_service.cc:1514] Processing DeleteTablet for tablet 3905631bc00442a38b40acacc5121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51504
I20260620 14:30:36.684134 6334 tablet_service.cc:1514] Processing DeleteTablet for tablet 3905631bc00442a38b40acacc5121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44534
I20260620 14:30:36.684155 6474 tablet_service.cc:1514] Processing DeleteTablet for tablet 3905631bc00442a38b40acacc5121556 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:36 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35984
I20260620 14:30:36.684331 6645 tablet_replica.cc:330] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: stopping tablet replica
I20260620 14:30:36.684352 6643 tablet_replica.cc:330] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: stopping tablet replica
I20260620 14:30:36.684391 6644 tablet_replica.cc:330] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: stopping tablet replica
I20260620 14:30:36.684453 6645 raft_consensus.cc:2229] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:36.684463 6643 raft_consensus.cc:2229] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:36.684494 6644 raft_consensus.cc:2229] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:36.684571 6643 raft_consensus.cc:2258] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.684583 6645 raft_consensus.cc:2258] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.684679 6644 raft_consensus.cc:2258] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:36.685199 6054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:30:36.685401 6643 ts_tablet_manager.cc:1905] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:36.685446 6645 ts_tablet_manager.cc:1905] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:30:36.685724 6054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:36.686935 6644 ts_tablet_manager.cc:1905] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:36.687703 6645 ts_tablet_manager.cc:1918] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:36.687773 6645 log.cc:1196] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/3905631bc00442a38b40acacc5121556
I20260620 14:30:36.687981 6645 ts_tablet_manager.cc:1939] T 3905631bc00442a38b40acacc5121556 P 9dcdb8ff9fe1400395929693cea76a01: Deleting consensus metadata
I20260620 14:30:36.687968 6643 ts_tablet_manager.cc:1918] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:36.688035 6643 log.cc:1196] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/3905631bc00442a38b40acacc5121556
I20260620 14:30:36.688267 6643 ts_tablet_manager.cc:1939] T 3905631bc00442a38b40acacc5121556 P 0711672969144a378fbc571906e0e304: Deleting consensus metadata
I20260620 14:30:36.689085 6042 catalog_manager.cc:4841] TS 0711672969144a378fbc571906e0e304 (127.4.58.66:44861): tablet 3905631bc00442a38b40acacc5121556 (table test-table [id=eaa4759c180742c89b03bb54a6957927]) successfully deleted
I20260620 14:30:36.689096 6043 catalog_manager.cc:4841] TS 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393): tablet 3905631bc00442a38b40acacc5121556 (table test-table [id=eaa4759c180742c89b03bb54a6957927]) successfully deleted
I20260620 14:30:36.689440 6644 ts_tablet_manager.cc:1918] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:30:36.689505 6644 log.cc:1196] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/3905631bc00442a38b40acacc5121556
I20260620 14:30:36.689699 6644 ts_tablet_manager.cc:1939] T 3905631bc00442a38b40acacc5121556 P c7c5f7ed4ca44f25b24ee6410bca29ed: Deleting consensus metadata
I20260620 14:30:36.690390 6043 catalog_manager.cc:4841] TS c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447): tablet 3905631bc00442a38b40acacc5121556 (table test-table [id=eaa4759c180742c89b03bb54a6957927]) successfully deleted
I20260620 14:30:36.692588 6334 tablet_service.cc:1467] Processing CreateTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab (DEFAULT_TABLE table=test-table [id=3f9887d926944b4fb3aa93db3f128b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.692585 6193 tablet_service.cc:1467] Processing CreateTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab (DEFAULT_TABLE table=test-table [id=3f9887d926944b4fb3aa93db3f128b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.692713 6474 tablet_service.cc:1467] Processing CreateTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab (DEFAULT_TABLE table=test-table [id=3f9887d926944b4fb3aa93db3f128b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:36.692864 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e810c3830894e338d4bd0bb6ca19dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.692863 6334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e810c3830894e338d4bd0bb6ca19dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.693012 6474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e810c3830894e338d4bd0bb6ca19dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:36.694728 6657 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Bootstrap starting.
I20260620 14:30:36.695041 6654 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Bootstrap starting.
I20260620 14:30:36.695092 6655 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed: Bootstrap starting.
I20260620 14:30:36.695708 6657 tablet_bootstrap.cc:654] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.695839 6654 tablet_bootstrap.cc:654] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.695878 6655 tablet_bootstrap.cc:654] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:36.696866 6655 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed: No bootstrap required, opened a new log
I20260620 14:30:36.696972 6655 ts_tablet_manager.cc:1397] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:36.697113 6657 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: No bootstrap required, opened a new log
I20260620 14:30:36.697193 6657 ts_tablet_manager.cc:1397] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:30:36.697320 6654 tablet_bootstrap.cc:492] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: No bootstrap required, opened a new log
I20260620 14:30:36.697402 6654 ts_tablet_manager.cc:1397] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:36.697374 6655 raft_consensus.cc:348] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.697484 6655 raft_consensus.cc:374] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.697526 6655 raft_consensus.cc:729] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.697608 6657 raft_consensus.cc:348] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.697652 6655 consensus_queue.cc:260] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.697708 6657 raft_consensus.cc:374] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.697746 6657 raft_consensus.cc:729] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcdb8ff9fe1400395929693cea76a01, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.697813 6654 raft_consensus.cc:348] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.697865 6657 consensus_queue.cc:260] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.697921 6654 raft_consensus.cc:374] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:36.697926 6655 ts_tablet_manager.cc:1428] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:36.697955 6654 raft_consensus.cc:729] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0711672969144a378fbc571906e0e304, State: Initialized, Role: FOLLOWER
I20260620 14:30:36.698105 6657 ts_tablet_manager.cc:1428] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:30:36.698074 6654 consensus_queue.cc:260] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.698331 6654 ts_tablet_manager.cc:1428] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:36.722081 6571 raft_consensus.cc:482] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:36.722251 6571 raft_consensus.cc:504] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.722702 6571 leader_election.cc:288] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:36.723132 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e810c3830894e338d4bd0bb6ca19dab" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01" is_pre_election: true
I20260620 14:30:36.723133 6353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e810c3830894e338d4bd0bb6ca19dab" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304" is_pre_election: true
I20260620 14:30:36.723284 6494 raft_consensus.cc:2454] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:36.723357 6353 raft_consensus.cc:2454] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 0.
I20260620 14:30:36.723618 6147 leader_election.cc:302] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:36.723810 6571 raft_consensus.cc:2781] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:36.723866 6571 raft_consensus.cc:482] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:36.723892 6571 raft_consensus.cc:3037] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.724865 6571 raft_consensus.cc:504] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.725208 6571 leader_election.cc:288] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Requested vote from peers 0711672969144a378fbc571906e0e304 (127.4.58.66:44861), 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393)
I20260620 14:30:36.725546 6353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e810c3830894e338d4bd0bb6ca19dab" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0711672969144a378fbc571906e0e304"
I20260620 14:30:36.725560 6494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e810c3830894e338d4bd0bb6ca19dab" candidate_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcdb8ff9fe1400395929693cea76a01"
I20260620 14:30:36.725661 6494 raft_consensus.cc:3037] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.725654 6353 raft_consensus.cc:3037] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:36.726557 6494 raft_consensus.cc:2454] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:36.726569 6353 raft_consensus.cc:2454] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7c5f7ed4ca44f25b24ee6410bca29ed in term 1.
I20260620 14:30:36.726814 6147 leader_election.cc:302] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcdb8ff9fe1400395929693cea76a01, c7c5f7ed4ca44f25b24ee6410bca29ed; no voters:
I20260620 14:30:36.726974 6571 raft_consensus.cc:2781] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:36.727059 6571 raft_consensus.cc:686] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [term 1 LEADER]: Becoming Leader. State: Replica: c7c5f7ed4ca44f25b24ee6410bca29ed, State: Running, Role: LEADER
I20260620 14:30:36.727197 6571 consensus_queue.cc:237] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } }
I20260620 14:30:36.728799 6056 catalog_manager.cc:5495] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65). New cstate: current_term: 1 leader_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7c5f7ed4ca44f25b24ee6410bca29ed" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:36.749496 6353 raft_consensus.cc:1261] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:36.749541 6493 raft_consensus.cc:1261] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Refusing update from remote peer c7c5f7ed4ca44f25b24ee6410bca29ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:36.749903 6625 consensus_queue.cc:1035] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0711672969144a378fbc571906e0e304" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:36.750079 6630 consensus_queue.cc:1035] T 6e810c3830894e338d4bd0bb6ca19dab P c7c5f7ed4ca44f25b24ee6410bca29ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcdb8ff9fe1400395929693cea76a01" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:30:36.763099 6623 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:30:38.312039 6052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:38.312191 6052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:38.314635 6052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f99bb414a34e46ff9a7347540100634c: Sending DeleteTablet for 3 replicas of tablet 6e810c3830894e338d4bd0bb6ca19dab
I20260620 14:30:38.315161 6334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:38 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44534
I20260620 14:30:38.315145 6193 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:38 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51504
I20260620 14:30:38.315218 6474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e810c3830894e338d4bd0bb6ca19dab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:38 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35984
I20260620 14:30:38.315507 6693 tablet_replica.cc:330] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: stopping tablet replica
I20260620 14:30:38.315616 6693 raft_consensus.cc:2229] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:38.315608 6695 tablet_replica.cc:330] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: stopping tablet replica
I20260620 14:30:38.315699 6693 raft_consensus.cc:2258] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:38.315724 6695 raft_consensus.cc:2229] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:38.315806 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6129
I20260620 14:30:38.315829 6695 raft_consensus.cc:2258] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:38.317476 6695 ts_tablet_manager.cc:1905] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:38.318100 6693 ts_tablet_manager.cc:1905] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:38.320122 6695 ts_tablet_manager.cc:1918] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:30:38.320204 6695 log.cc:1196] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/6e810c3830894e338d4bd0bb6ca19dab
I20260620 14:30:38.320178 6693 ts_tablet_manager.cc:1918] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:30:38.320248 6693 log.cc:1196] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/6e810c3830894e338d4bd0bb6ca19dab
I20260620 14:30:38.320458 6695 ts_tablet_manager.cc:1939] T 6e810c3830894e338d4bd0bb6ca19dab P 9dcdb8ff9fe1400395929693cea76a01: Deleting consensus metadata
I20260620 14:30:38.320493 6693 ts_tablet_manager.cc:1939] T 6e810c3830894e338d4bd0bb6ca19dab P 0711672969144a378fbc571906e0e304: Deleting consensus metadata
I20260620 14:30:38.321234 6042 catalog_manager.cc:4841] TS 0711672969144a378fbc571906e0e304 (127.4.58.66:44861): tablet 6e810c3830894e338d4bd0bb6ca19dab (table test-table [id=3f9887d926944b4fb3aa93db3f128b1c]) successfully deleted
I20260620 14:30:38.321283 6043 catalog_manager.cc:4841] TS 9dcdb8ff9fe1400395929693cea76a01 (127.4.58.67:33393): tablet 6e810c3830894e338d4bd0bb6ca19dab (table test-table [id=3f9887d926944b4fb3aa93db3f128b1c]) successfully deleted
W20260620 14:30:38.323802 6043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:46447 (error 108)
I20260620 14:30:38.324044 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6269
I20260620 14:30:38.325743 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:38.325214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:46447 (local address 127.0.0.1:58020)
0620 14:30:38.325576 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:38.325650 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:46447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
W20260620 14:30:38.325877 6043 catalog_manager.cc:4584] TS c7c5f7ed4ca44f25b24ee6410bca29ed (127.4.58.65:46447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e810c3830894e338d4bd0bb6ca19dab: Network error: Client connection negotiation failed: client connection to 127.4.58.65:46447: connect: Connection refused (error 111)
I20260620 14:30:38.330129 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6409
I20260620 14:30:38.335132 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6025
2026-06-20T14:30:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6231 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:38 dist-test-slave-gqzk krb5kdc[6700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:38 dist-test-slave-gqzk krb5kdc[6700](info): set up 2 sockets
Jun 20 14:30:38 dist-test-slave-gqzk krb5kdc[6700](info): commencing operation
krb5kdc: starting...
W20260620 14:30:39.296625 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:39 dist-test-slave-gqzk krb5kdc[6700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:39.416002 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42475
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:42475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:39.527338 6713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:39.527633 6713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:39.527673 6713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:39.531677 6713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:39.531741 6713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:39.531759 6713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:39.531774 6713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:39.531788 6713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:39.537631 6713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:42475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:39.538710 6713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:39.539717 6713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:39.551064 6718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:39.551049 6721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:39.551065 6713 server_base.cc:1034] running on GCE node
W20260620 14:30:39.551267 6719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:39.552048 6713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:39.553469 6713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:39.554646 6713 hybrid_clock.cc:648] HybridClock initialized: now 1781965839554613 us; error 49 us; skew 500 ppm
Jun 20 14:30:39 dist-test-slave-gqzk krb5kdc[6700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965839, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:39.557797 6713 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:39.558714 6713 webserver.cc:460] Webserver started at http://127.4.58.126:43581/ using document root <none> and password file <none>
I20260620 14:30:39.559259 6713 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:39.559329 6713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:39.559537 6713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:39.562192 6713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "d7dd8d26ed1341c78aa61ffb1c983668"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "65a49691b09eb2dc2cbefe287721739d"
server_key_iv: "3632c1b5ce67b02da2d87b9c209306d9"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.562945 6713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "d7dd8d26ed1341c78aa61ffb1c983668"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "65a49691b09eb2dc2cbefe287721739d"
server_key_iv: "3632c1b5ce67b02da2d87b9c209306d9"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.568517 6713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 14:30:39.571650 6728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:39.573191 6713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:39.573330 6713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "d7dd8d26ed1341c78aa61ffb1c983668"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "65a49691b09eb2dc2cbefe287721739d"
server_key_iv: "3632c1b5ce67b02da2d87b9c209306d9"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.573462 6713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:39.596196 6713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:39.599548 6713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:39.599712 6713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:39.606904 6713 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:42475
I20260620 14:30:39.606931 6780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:42475 every 8 connection(s)
I20260620 14:30:39.607924 6713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:39.610862 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:39.612639 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6713
I20260620 14:30:39.612730 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:39.613011 4329 external_mini_cluster.cc:1428] Setting key 4f8ebcbb9ab498f60694d4025d0b59b7
I20260620 14:30:39.617502 6781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668: Bootstrap starting.
I20260620 14:30:39.619818 6781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:39.620551 6781 log.cc:824] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:30:39 dist-test-slave-gqzk krb5kdc[6700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:39.622521 6781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668: No bootstrap required, opened a new log
I20260620 14:30:39.626178 6781 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } }
I20260620 14:30:39.626427 6781 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:39.626463 6781 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dd8d26ed1341c78aa61ffb1c983668, State: Initialized, Role: FOLLOWER
I20260620 14:30:39.626823 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:39.614495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.4.58.126:42475)
0620 14:30:39.614992 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:39.615009 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:39.615055 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:30:39.615776 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:39.615795 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:39.616149 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:39.616478 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:39.617355 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.618354 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.619251 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.619736 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.619885 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:39.622266 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:39.622271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:39.622286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:39.622314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:39.623952 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:39.624368 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:39.624373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:39.624378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:39.624499 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:39.624778 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:39.624782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:39.624784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:39.625195 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:39.625407 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:39.625699 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:30:39.625981 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260620 14:30:39.626912 6781 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } }
I20260620 14:30:39.627067 6781 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:39.627120 6781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:39.627192 6781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:39.628131 6781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } }
I20260620 14:30:39.628505 6781 leader_election.cc:302] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7dd8d26ed1341c78aa61ffb1c983668; no voters:
I20260620 14:30:39.628893 6781 leader_election.cc:288] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:39.628966 6786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:39.629171 6786 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [term 1 LEADER]: Becoming Leader. State: Replica: d7dd8d26ed1341c78aa61ffb1c983668, State: Running, Role: LEADER
I20260620 14:30:39.629483 6786 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } }
I20260620 14:30:39.630105 6781 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:39.631052 6787 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7dd8d26ed1341c78aa61ffb1c983668. Latest consensus state: current_term: 1 leader_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } } }
I20260620 14:30:39.631178 6787 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:39.631474 6788 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dd8d26ed1341c78aa61ffb1c983668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42475 } } }
I20260620 14:30:39.631561 6788 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:39.631671 6795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:39.634570 6795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:39.640137 6795 catalog_manager.cc:1347] Generated new cluster ID: 3c4fd4b4ab044b67be992232f39a5cf2
I20260620 14:30:39.640218 6795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:39.670264 6795 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:39.671308 6795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:39.683634 6795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7dd8d26ed1341c78aa61ffb1c983668: Generated new TSK 0
I20260620 14:30:39.684576 6795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:39.720439 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:39.835248 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:39.835569 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:39.835638 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:39.839810 6807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:39.839903 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:39.840003 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:39.845515 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:39.846683 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:39.847740 6807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:39.857772 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:39.857812 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:39.857867 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:39.858168 6807 server_base.cc:1034] running on GCE node
I20260620 14:30:39.858594 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:39.859237 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:39.860410 6807 hybrid_clock.cc:648] HybridClock initialized: now 1781965839860387 us; error 33 us; skew 500 ppm
Jun 20 14:30:39 dist-test-slave-gqzk krb5kdc[6700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965839, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:39.863085 6807 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:39.863979 6807 webserver.cc:460] Webserver started at http://127.4.58.65:42101/ using document root <none> and password file <none>
I20260620 14:30:39.864506 6807 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:39.864564 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:39.864791 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:39.866889 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "a443073404c642cf88c068cef81b77a4"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "ed0c9e40ce7275d90b5712c6d7500301"
server_key_iv: "b5e30efaea8d72196ce122fb8fed6505"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.867410 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "a443073404c642cf88c068cef81b77a4"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "ed0c9e40ce7275d90b5712c6d7500301"
server_key_iv: "b5e30efaea8d72196ce122fb8fed6505"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.871270 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:39.874056 6822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:39.875393 6807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:39.875509 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "a443073404c642cf88c068cef81b77a4"
format_stamp: "Formatted at 2026-06-20 14:30:39 on dist-test-slave-gqzk"
server_key: "ed0c9e40ce7275d90b5712c6d7500301"
server_key_iv: "b5e30efaea8d72196ce122fb8fed6505"
server_key_version: "encryptionkey@0"
I20260620 14:30:39.875602 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:39.886122 6807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:39.889302 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:39.889472 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:39.890004 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:39.890808 6807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:39.890856 6807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:39.890897 6807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:39.890909 6807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:39.899466 6807 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:44143
I20260620 14:30:39.899495 6935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:44143 every 8 connection(s)
I20260620 14:30:39.900396 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:39.907804 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6807
I20260620 14:30:39.907910 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:39.908155 4329 external_mini_cluster.cc:1428] Setting key c726b46ae4585ff3217d38ecfd7a292b
Jun 20 14:30:39 dist-test-slave-gqzk krb5kdc[6700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965839, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:39.915248 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:39.902257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:45739 (local address 127.4.58.126:42475)
0620 14:30:39.902430 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:39.902434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:39.903278 (+ 844us) server_negotiation.cc:407] Connection header received
0620 14:30:39.904651 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:39.904654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:39.904712 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:39.904851 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:39.906007 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.906532 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.907680 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.908090 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.908188 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:39.911360 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:39.911365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:39.911367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:39.911396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:39.912938 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:39.913544 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:39.913548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:39.913550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:39.913662 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:39.914049 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:39.914051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:39.914052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:39.914234 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:39.914362 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:39.915007 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 14:30:39.915124 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:30:39.916417 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:39.902635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42475 (local address 127.4.58.65:45739)
0620 14:30:39.903120 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:39.903165 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:39.904443 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:39.905008 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:39.905014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:39.905392 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:39.905868 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:39.905882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.906645 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.906648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:39.907552 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:39.907563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:39.908276 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:39.908290 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:39.908486 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:39.909465 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:39.909481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:39.911181 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:39.913044 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:39.913051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:39.913069 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:39.913422 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:39.913756 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:39.913767 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:39.913770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:39.913958 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:39.914341 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:39.914349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:39.914699 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 14:30:39.914940 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:30:39.915318 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260620 14:30:39.917743 6936 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42475
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:39.917989 6936 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:39.918515 6936 heartbeater.cc:511] Master 127.4.58.126:42475 requested a full tablet report, sending...
I20260620 14:30:39.920181 6745 ts_manager.cc:194] Registered new tserver with Master: a443073404c642cf88c068cef81b77a4 (127.4.58.65:44143)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:39.921360 6745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:45739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:39.935371 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:40.051376 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:40.051678 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:40.051723 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:40.056073 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:40.056164 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:40.056239 6941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:40.062420 6941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:40.063581 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:40.064666 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:40.074920 6941 server_base.cc:1034] running on GCE node
W20260620 14:30:40.074971 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:40.074936 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:40.074941 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:40.075647 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:40.076318 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:40.077484 6941 hybrid_clock.cc:648] HybridClock initialized: now 1781965840077464 us; error 37 us; skew 500 ppm
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[6700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:40.080309 6941 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:40.081282 6941 webserver.cc:460] Webserver started at http://127.4.58.66:36339/ using document root <none> and password file <none>
I20260620 14:30:40.081843 6941 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:40.081911 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:40.082108 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:40.084195 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "653893d6b576423395dc43e43cf12b69"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "6699cd38a73eea0194edcecde13c30b8"
server_key_iv: "bb21aa87a306dbb5b559d9f5d5f8d3ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.084800 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "653893d6b576423395dc43e43cf12b69"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "6699cd38a73eea0194edcecde13c30b8"
server_key_iv: "bb21aa87a306dbb5b559d9f5d5f8d3ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.088716 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:30:40.091521 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.093012 6941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:40.093146 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "653893d6b576423395dc43e43cf12b69"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "6699cd38a73eea0194edcecde13c30b8"
server_key_iv: "bb21aa87a306dbb5b559d9f5d5f8d3ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.093258 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:40.116307 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:40.119752 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:40.119943 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:40.120554 6941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:40.121732 6941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:40.121796 6941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.121856 6941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:40.121886 6941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.131016 6941 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:40401
I20260620 14:30:40.131074 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:40401 every 8 connection(s)
I20260620 14:30:40.132005 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:40.137979 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 6941
I20260620 14:30:40.138113 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:40.138401 4329 external_mini_cluster.cc:1428] Setting key 4cb3e7128d14c02bbec7e4e7cb161a92
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[6700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:40.147905 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:40.133892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:35103 (local address 127.4.58.126:42475)
0620 14:30:40.134030 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:40.134034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:40.135046 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 14:30:40.136523 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:40.136526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:40.136575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:40.136656 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:40.138212 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.138931 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.140474 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.140835 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.140943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:40.143663 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:40.143667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:40.143670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:40.143699 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:40.145259 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:40.145973 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:40.145977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:40.145978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:40.146069 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:40.146544 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:40.146549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:40.146550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:40.146808 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:40.146906 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:40.147623 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 14:30:40.147772 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:40.149173 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:40.134331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42475 (local address 127.4.58.66:35103)
0620 14:30:40.134842 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:40.134893 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:40.136280 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:40.136883 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:40.136891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:40.137355 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:40.137995 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:40.138018 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.139098 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.139102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:40.140306 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:40.140317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.140978 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.140981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:40.141134 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:40.141778 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:40.141801 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:40.143482 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:40.145399 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:40.145407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:40.145427 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:40.145848 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:40.146163 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:40.146166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:40.146168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:40.146406 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:40.146924 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:40.146932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:40.147313 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 14:30:40.147552 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 14:30:40.147949 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260620 14:30:40.150532 7070 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42475
I20260620 14:30:40.150785 7070 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:40.151353 7070 heartbeater.cc:511] Master 127.4.58.126:42475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:30:40.152596 6745 ts_manager.cc:194] Registered new tserver with Master: 653893d6b576423395dc43e43cf12b69 (127.4.58.66:40401)
I20260620 14:30:40.153368 6745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:35103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:40.168111 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:40.281515 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:40.281798 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:40.281842 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:40.285900 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:40.285971 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:40.286032 7075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:40.291359 7075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:40.292467 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:40.293648 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:40.303510 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:40.303496 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:40.303534 7075 server_base.cc:1034] running on GCE node
W20260620 14:30:40.303496 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:40.304172 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:40.304819 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:40.305982 7075 hybrid_clock.cc:648] HybridClock initialized: now 1781965840305963 us; error 31 us; skew 500 ppm
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[6700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:40.308697 7075 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:40.309609 7075 webserver.cc:460] Webserver started at http://127.4.58.67:40495/ using document root <none> and password file <none>
I20260620 14:30:40.310150 7075 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:40.310218 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:40.310405 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:40.312479 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "a22d0afb65834173a11bca23f836e3aa"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "0f87ca2edadcf65f7592a3165e0212a2"
server_key_iv: "6a22d505d046a9440f4f0727384caded"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.313025 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "a22d0afb65834173a11bca23f836e3aa"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "0f87ca2edadcf65f7592a3165e0212a2"
server_key_iv: "6a22d505d046a9440f4f0727384caded"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.316582 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:30:40.319005 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.320201 7075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:40.320308 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "a22d0afb65834173a11bca23f836e3aa"
format_stamp: "Formatted at 2026-06-20 14:30:40 on dist-test-slave-gqzk"
server_key: "0f87ca2edadcf65f7592a3165e0212a2"
server_key_iv: "6a22d505d046a9440f4f0727384caded"
server_key_version: "encryptionkey@0"
I20260620 14:30:40.320403 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:40.345206 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:40.348485 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:40.348660 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:40.349287 7075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:40.350121 7075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:40.350167 7075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.350221 7075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:40.350251 7075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:40.359916 7075 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:35667
I20260620 14:30:40.359957 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:35667 every 8 connection(s)
I20260620 14:30:40.360859 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:40.365036 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7075
I20260620 14:30:40.365135 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:40.365379 4329 external_mini_cluster.cc:1428] Setting key 25ade004f0f6dc755fb8893c74283888
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[6700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:40.374143 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:40.362696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:45099 (local address 127.4.58.126:42475)
0620 14:30:40.362827 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:40.362831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:40.363663 (+ 832us) server_negotiation.cc:407] Connection header received
0620 14:30:40.365012 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:40.365015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:40.365074 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:40.365146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:40.366499 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.367024 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.368128 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.368336 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.368422 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:40.370800 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:40.370803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:40.370805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:40.370830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:40.372055 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:40.372551 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:40.372554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:40.372555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:40.372626 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:40.372978 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:40.372981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:40.372983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:40.373166 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:40.373250 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:40.373870 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 14:30:40.373985 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:30:40.374966 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:40.363070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42475 (local address 127.4.58.67:45099)
0620 14:30:40.363515 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:40.363557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:40.364797 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:40.365326 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:40.365333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:40.365717 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:40.366350 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:40.366378 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.367139 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.367142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:40.368012 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:40.368021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:40.368470 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:40.368474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:40.368631 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:40.369165 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:40.369182 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:40.370646 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:40.372176 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:40.372182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:40.372198 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:40.372454 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:40.372707 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:40.372710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:40.372712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:40.372891 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:40.373315 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:40.373321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:40.373597 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 14:30:40.373799 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 14:30:40.374134 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260620 14:30:40.376181 7204 heartbeater.cc:347] Connected to a master server at 127.4.58.126:42475
I20260620 14:30:40.376475 7204 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:40.377023 7204 heartbeater.cc:511] Master 127.4.58.126:42475 requested a full tablet report, sending...
I20260620 14:30:40.378139 6745 ts_manager.cc:194] Registered new tserver with Master: a22d0afb65834173a11bca23f836e3aa (127.4.58.67:35667)
I20260620 14:30:40.378829 6745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:45099
I20260620 14:30:40.379506 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:30:40.391992 6784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:40.390477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.4.58.126:42475)
0620 14:30:40.390640 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:40.390644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:40.390784 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:30:40.390889 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:40.390892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:40.390938 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:40.391014 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:40.391854 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56416: BlockingRecv error: recv got EOF from 127.0.0.1:56416 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:30:40.393249 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6807
I20260620 14:30:40.398102 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6941
I20260620 14:30:40.402947 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7075
I20260620 14:30:40.407038 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 6713
2026-06-20T14:30:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2070 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[7216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[7216](info): set up 2 sockets
Jun 20 14:30:40 dist-test-slave-gqzk krb5kdc[7216](info): commencing operation
krb5kdc: starting...
W20260620 14:30:41.360711 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:41.480135 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46841
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:41.592305 7229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:41.592602 7229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:41.592659 7229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:41.596817 7229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:41.596894 7229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:41.596920 7229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:41.596961 7229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:41.596992 7229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:41.602428 7229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:41.603538 7229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:41.604576 7229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:41.612910 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:41.612890 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:41.612931 7229 server_base.cc:1034] running on GCE node
W20260620 14:30:41.612890 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:41.613781 7229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:41.614935 7229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:41.616112 7229 hybrid_clock.cc:648] HybridClock initialized: now 1781965841616087 us; error 37 us; skew 500 ppm
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965841, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:41.625602 7229 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:41.626614 7229 webserver.cc:460] Webserver started at http://127.4.58.126:40443/ using document root <none> and password file <none>
I20260620 14:30:41.627172 7229 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:41.627236 7229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:41.627429 7229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:41.629635 7229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "362f32cb184244d284ce1b094d8b9ece"
format_stamp: "Formatted at 2026-06-20 14:30:41 on dist-test-slave-gqzk"
server_key: "5746bb58547b3515586a6e9e77c1f97e"
server_key_iv: "9fe4ddd83b5a1c81780f043494cb38e6"
server_key_version: "encryptionkey@0"
I20260620 14:30:41.630191 7229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "362f32cb184244d284ce1b094d8b9ece"
format_stamp: "Formatted at 2026-06-20 14:30:41 on dist-test-slave-gqzk"
server_key: "5746bb58547b3515586a6e9e77c1f97e"
server_key_iv: "9fe4ddd83b5a1c81780f043494cb38e6"
server_key_version: "encryptionkey@0"
I20260620 14:30:41.634011 7229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:41.636445 7244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:41.637810 7229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:41.637946 7229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "362f32cb184244d284ce1b094d8b9ece"
format_stamp: "Formatted at 2026-06-20 14:30:41 on dist-test-slave-gqzk"
server_key: "5746bb58547b3515586a6e9e77c1f97e"
server_key_iv: "9fe4ddd83b5a1c81780f043494cb38e6"
server_key_version: "encryptionkey@0"
I20260620 14:30:41.638046 7229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:41.648344 7229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:41.651034 7229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:41.651203 7229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:41.658118 7229 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:46841
I20260620 14:30:41.658509 7296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:46841 every 8 connection(s)
I20260620 14:30:41.659159 7229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:41.661966 7297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:41.666239 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7229
I20260620 14:30:41.666344 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:41.666622 4329 external_mini_cluster.cc:1428] Setting key 7d6c91727e511f3f724044b45debd354
I20260620 14:30:41.667843 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: Bootstrap starting.
I20260620 14:30:41.670531 7297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:41.671402 7297 log.cc:824] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:41.673431 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: No bootstrap required, opened a new log
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:41.677706 7297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } }
I20260620 14:30:41.678009 7297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:41.678066 7297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362f32cb184244d284ce1b094d8b9ece, State: Initialized, Role: FOLLOWER
I20260620 14:30:41.678526 7297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } }
I20260620 14:30:41.678668 7297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:41.678738 7297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:41.678844 7297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:41.679973 7297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } }
I20260620 14:30:41.680328 7297 leader_election.cc:302] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362f32cb184244d284ce1b094d8b9ece; no voters:
I20260620 14:30:41.680644 7297 leader_election.cc:288] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:41.680787 7302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:41.680984 7302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [term 1 LEADER]: Becoming Leader. State: Replica: 362f32cb184244d284ce1b094d8b9ece, State: Running, Role: LEADER
I20260620 14:30:41.681342 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:41.667942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.4.58.126:46841)
0620 14:30:41.668381 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:41.668400 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:41.668440 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:30:41.669274 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:41.669297 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:41.669718 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:41.670078 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:41.670709 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:41.672026 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:41.672891 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:41.673568 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:41.673720 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:41.676240 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:41.676248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:41.676268 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:41.676303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:41.678219 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:41.678651 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:41.678657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:41.678663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:41.678806 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:41.679086 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:41.679090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:41.679091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:41.679608 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:41.679812 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:41.679992 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:30:41.680302 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":113,"threads_started":1}
I20260620 14:30:41.681339 7302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } }
I20260620 14:30:41.681924 7297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:41.682928 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "362f32cb184244d284ce1b094d8b9ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } } }
I20260620 14:30:41.683043 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:41.683414 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362f32cb184244d284ce1b094d8b9ece. Latest consensus state: current_term: 1 leader_uuid: "362f32cb184244d284ce1b094d8b9ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362f32cb184244d284ce1b094d8b9ece" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46841 } } }
I20260620 14:30:41.683557 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:41.684597 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:30:41.686852 7317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:41.686931 7317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:41.686993 7311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:41.687633 7311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:41.693017 7311 catalog_manager.cc:1347] Generated new cluster ID: 1db3325993554bcbb76c50f922c688b3
I20260620 14:30:41.693078 7311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:41.709091 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:41.704524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.4.58.126:46841)
0620 14:30:41.704646 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:41.704651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:41.704935 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:30:41.705071 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:41.705074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:41.705118 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:41.705185 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:41.705705 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:41.706215 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:41.707037 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:41.707280 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:41.707379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:41.708134 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:41.708138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:41.708140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:41.708608 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:41.708638 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:41.708653 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:41.708732 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:41.708873 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:30:41.708940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:41.713430 7311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:41.714326 7311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:41.719970 7311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 362f32cb184244d284ce1b094d8b9ece: Generated new TSK 0
I20260620 14:30:41.720606 7311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:41.726749 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7229
2026-06-20T14:30:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1318 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7330](info): set up 2 sockets
Jun 20 14:30:41 dist-test-slave-gqzk krb5kdc[7330](info): commencing operation
krb5kdc: starting...
W20260620 14:30:42.672498 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:42 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:42.791311 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44541
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:42.902915 7343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:42.903213 7343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:42.903261 7343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:42.907445 7343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:42.907513 7343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:42.907532 7343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:42.907548 7343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:42.907562 7343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:42.913079 7343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:42.914283 7343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:42.915344 7343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:42.923928 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:42.923933 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:42.924120 7343 server_base.cc:1034] running on GCE node
W20260620 14:30:42.923928 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:42.924914 7343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:42.926133 7343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:42.927323 7343 hybrid_clock.cc:648] HybridClock initialized: now 1781965842927291 us; error 43 us; skew 500 ppm
Jun 20 14:30:42 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:42.930256 7343 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:42.931252 7343 webserver.cc:460] Webserver started at http://127.4.58.126:46723/ using document root <none> and password file <none>
I20260620 14:30:42.931876 7343 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:42.931948 7343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:42.932153 7343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:42.934284 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23"
format_stamp: "Formatted at 2026-06-20 14:30:42 on dist-test-slave-gqzk"
server_key: "9bea4a0bd7ef251db457341c7b8d927f"
server_key_iv: "fd8f495deafd1f0ae5305332371cf806"
server_key_version: "encryptionkey@0"
I20260620 14:30:42.934852 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23"
format_stamp: "Formatted at 2026-06-20 14:30:42 on dist-test-slave-gqzk"
server_key: "9bea4a0bd7ef251db457341c7b8d927f"
server_key_iv: "fd8f495deafd1f0ae5305332371cf806"
server_key_version: "encryptionkey@0"
I20260620 14:30:42.938697 7343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:42.941218 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:42.942472 7343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:42.942590 7343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23"
format_stamp: "Formatted at 2026-06-20 14:30:42 on dist-test-slave-gqzk"
server_key: "9bea4a0bd7ef251db457341c7b8d927f"
server_key_iv: "fd8f495deafd1f0ae5305332371cf806"
server_key_version: "encryptionkey@0"
I20260620 14:30:42.942690 7343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:42.977342 7343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:42.980528 7343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:42.980685 7343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:42.987571 7343 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44541
I20260620 14:30:42.987867 7410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44541 every 8 connection(s)
I20260620 14:30:42.988658 7343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:42.991559 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:42.997252 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: Bootstrap starting.
I20260620 14:30:42.997993 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7343
I20260620 14:30:42.998088 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:42.998334 4329 external_mini_cluster.cc:1428] Setting key b1c06021fdc50f379e7d1e3651a7b855
I20260620 14:30:43.000067 7411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:43.000815 7411 log.cc:824] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:43.002902 7411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: No bootstrap required, opened a new log
I20260620 14:30:43.006428 7411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } }
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:43.006752 7411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:43.006807 7411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf2ec03d48343eaacc06b0ee2bb9c23, State: Initialized, Role: FOLLOWER
I20260620 14:30:43.007288 7411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } }
I20260620 14:30:43.007433 7411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:43.007498 7411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:43.007597 7411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:43.008483 7411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } }
I20260620 14:30:43.008847 7411 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cf2ec03d48343eaacc06b0ee2bb9c23; no voters:
I20260620 14:30:43.009166 7411 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:43.009277 7416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:43.009485 7416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [term 1 LEADER]: Becoming Leader. State: Replica: 3cf2ec03d48343eaacc06b0ee2bb9c23, State: Running, Role: LEADER
I20260620 14:30:43.009891 7416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } }
I20260620 14:30:43.010433 7411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:43.011088 7418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf2ec03d48343eaacc06b0ee2bb9c23. Latest consensus state: current_term: 1 leader_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } } }
I20260620 14:30:43.011207 7418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:43.011890 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf2ec03d48343eaacc06b0ee2bb9c23" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44541 } } }
I20260620 14:30:43.011972 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:43.013222 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:42.999545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.4.58.126:44541)
0620 14:30:43.000015 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:43.000029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:43.000066 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:43.000776 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:43.000803 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:43.001185 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:43.001510 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:43.002286 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.003319 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.004159 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.004636 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.004806 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.007361 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:43.007367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:43.007381 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:43.007412 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:43.009350 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.009902 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.009908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.009915 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.010058 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.010421 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.010425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.010426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.010993 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:43.011180 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:43.011850 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 14:30:43.012174 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
W20260620 14:30:43.015228 7431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:43.015314 7431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:43.015368 7424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:43.016157 7424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:43.021704 7424 catalog_manager.cc:1347] Generated new cluster ID: 2b535a5edef24546bb97d0e5d9db0a13
I20260620 14:30:43.021786 7424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:43.078799 7424 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:43.080123 7424 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:43.095012 7424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cf2ec03d48343eaacc06b0ee2bb9c23: Generated new TSK 0
I20260620 14:30:43.095753 7424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:43.122104 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:43.232295 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:43.232583 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:43.232643 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:43.236608 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:43.236675 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:43.236783 7437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:43.242029 7437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:43.243064 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:43.244071 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:43.253589 7445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:43.253705 7437 server_base.cc:1034] running on GCE node
W20260620 14:30:43.253595 7443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:43.253584 7442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:43.254334 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:43.254977 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:43.256155 7437 hybrid_clock.cc:648] HybridClock initialized: now 1781965843256132 us; error 32 us; skew 500 ppm
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:43.258909 7437 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:43.259747 7437 webserver.cc:460] Webserver started at http://127.4.58.65:45307/ using document root <none> and password file <none>
I20260620 14:30:43.260288 7437 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:43.260353 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:43.260535 7437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:43.262595 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "ae9355bbd1b34b96a2ae3a5624923765"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "767e3df4b60da9a02a5341fbbf7d97c1"
server_key_iv: "6d5c80661f6d0ebb3f2a67b4436aa4df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.263152 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9355bbd1b34b96a2ae3a5624923765"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "767e3df4b60da9a02a5341fbbf7d97c1"
server_key_iv: "6d5c80661f6d0ebb3f2a67b4436aa4df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.266781 7437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:30:43.269272 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.270524 7437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:43.270627 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "ae9355bbd1b34b96a2ae3a5624923765"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "767e3df4b60da9a02a5341fbbf7d97c1"
server_key_iv: "6d5c80661f6d0ebb3f2a67b4436aa4df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.270720 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:43.288100 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:43.291404 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:43.291560 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:43.292153 7437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:43.292995 7437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:43.293051 7437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.293102 7437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:43.293123 7437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.301852 7437 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:36055
I20260620 14:30:43.301882 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:36055 every 8 connection(s)
I20260620 14:30:43.302731 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:43.308350 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7437
I20260620 14:30:43.308476 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:43.308784 4329 external_mini_cluster.cc:1428] Setting key 5c5417de9c27838a00796bd19557bdeb
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:43.316694 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.304475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:52173 (local address 127.4.58.126:44541)
0620 14:30:43.304599 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:43.304602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:43.305444 (+ 842us) server_negotiation.cc:407] Connection header received
0620 14:30:43.306763 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:43.306766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:43.306837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:43.306912 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:43.308101 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.308896 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.310150 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.310455 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.310563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.313145 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:43.313148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:43.313151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:43.313175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:43.314555 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.315156 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.315158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.315159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.315229 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.315606 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.315609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.315610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.315792 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:43.315885 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:43.316474 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 14:30:43.316572 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:43.317910 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.304858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44541 (local address 127.4.58.65:52173)
0620 14:30:43.305281 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:43.305323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:43.306552 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:43.307074 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:43.307080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:43.307433 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:43.307939 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.307952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.309010 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.309012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.310042 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.310053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.310589 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.310592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.310729 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.311329 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:43.311346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:43.312991 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:43.314670 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.314677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.314694 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.315062 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.315305 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.315308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.315310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.315489 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.315912 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:43.315919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:43.316233 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 14:30:43.316444 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 14:30:43.316810 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:43.319175 7566 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44541
I20260620 14:30:43.319442 7566 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:43.319973 7566 heartbeater.cc:511] Master 127.4.58.126:44541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:43.321512 7375 ts_manager.cc:194] Registered new tserver with Master: ae9355bbd1b34b96a2ae3a5624923765 (127.4.58.65:36055)
I20260620 14:30:43.322615 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:52173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:43.333277 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:43.442096 7571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:43.442390 7571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:43.442437 7571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:43.446481 7571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:43.446556 7571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:43.446623 7571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:43.452095 7571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:43.453217 7571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:43.454186 7571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:43.463872 7576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:43.463879 7579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:43.463879 7577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:43.464322 7571 server_base.cc:1034] running on GCE node
I20260620 14:30:43.464679 7571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:43.465365 7571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:43.466542 7571 hybrid_clock.cc:648] HybridClock initialized: now 1781965843466510 us; error 50 us; skew 500 ppm
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:43.469411 7571 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:43.470321 7571 webserver.cc:460] Webserver started at http://127.4.58.66:36039/ using document root <none> and password file <none>
I20260620 14:30:43.470882 7571 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:43.470947 7571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:43.471133 7571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:43.473265 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "c68db88bdbd640b189edab0afe27bc92"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "bccd243d5378169818a68bff88ccd3aa"
server_key_iv: "7a36590eb1311571ca7f9c9df6d9a8df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.473825 7571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "c68db88bdbd640b189edab0afe27bc92"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "bccd243d5378169818a68bff88ccd3aa"
server_key_iv: "7a36590eb1311571ca7f9c9df6d9a8df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.477734 7571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:30:43.480177 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.481478 7571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:43.481585 7571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "c68db88bdbd640b189edab0afe27bc92"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "bccd243d5378169818a68bff88ccd3aa"
server_key_iv: "7a36590eb1311571ca7f9c9df6d9a8df"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.481678 7571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:43.499106 7571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:43.502902 7571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:43.503093 7571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:43.503713 7571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:43.504565 7571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:43.504612 7571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.504664 7571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:43.504693 7571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.513484 7571 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:45737
I20260620 14:30:43.513511 7699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:45737 every 8 connection(s)
I20260620 14:30:43.514364 7571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:43.519212 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7571
I20260620 14:30:43.519295 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:43.519531 4329 external_mini_cluster.cc:1428] Setting key 96e70e1779523cb2328ca1d5a2e6f980
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:43.528236 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.516070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:47387 (local address 127.4.58.126:44541)
0620 14:30:43.516192 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:43.516195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:43.517118 (+ 923us) server_negotiation.cc:407] Connection header received
0620 14:30:43.518469 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:43.518472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:43.518529 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:43.518600 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:43.519771 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.520348 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.521935 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.522154 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.522234 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.524670 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:43.524674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:43.524677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:43.524706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:43.526247 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.526743 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.526747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.526749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.526852 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.527149 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.527153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.527155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.527383 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:43.527484 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:43.527944 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 14:30:43.528072 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:43.529078 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.516514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44541 (local address 127.4.58.66:47387)
0620 14:30:43.516979 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:43.517021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:43.518238 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:43.518773 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:43.518781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:43.519122 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:43.519642 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.519656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.520466 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.520469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.521810 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.521822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.522288 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.522292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.522488 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.523078 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:43.523094 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:43.524504 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:43.526366 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.526372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.526385 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.526649 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.526933 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.526935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.526936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.527069 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.527472 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:43.527477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:43.527708 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 14:30:43.527860 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:30:43.528144 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":106,"threads_started":1}
I20260620 14:30:43.530210 7700 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44541
I20260620 14:30:43.530463 7700 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:30:43.530944 7700 heartbeater.cc:511] Master 127.4.58.126:44541 requested a full tablet report, sending...
I20260620 14:30:43.531930 7375 ts_manager.cc:194] Registered new tserver with Master: c68db88bdbd640b189edab0afe27bc92 (127.4.58.66:45737)
I20260620 14:30:43.532668 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:47387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:43.544736 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:43.656144 7705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:43.656417 7705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:43.656464 7705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:43.660557 7705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:43.660614 7705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:43.660672 7705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:43.666131 7705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:43.667145 7705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:43.668103 7705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:43.677413 7710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:43.677486 7713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:43.677470 7711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:43.677763 7705 server_base.cc:1034] running on GCE node
I20260620 14:30:43.678179 7705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:43.678885 7705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:43.680065 7705 hybrid_clock.cc:648] HybridClock initialized: now 1781965843680049 us; error 31 us; skew 500 ppm
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:43.682725 7705 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:43.683596 7705 webserver.cc:460] Webserver started at http://127.4.58.67:34057/ using document root <none> and password file <none>
I20260620 14:30:43.684132 7705 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:43.684196 7705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:43.684407 7705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:43.686750 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "6debe7a0f8b246aba3ed081225eb0123"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "d9b0d038d38ddbca27c2ef9d84485631"
server_key_iv: "e40449258e05a118d1c380d6c83559d0"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.687332 7705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "6debe7a0f8b246aba3ed081225eb0123"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "d9b0d038d38ddbca27c2ef9d84485631"
server_key_iv: "e40449258e05a118d1c380d6c83559d0"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.691030 7705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 14:30:43.693447 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.694689 7705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:43.694802 7705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "6debe7a0f8b246aba3ed081225eb0123"
format_stamp: "Formatted at 2026-06-20 14:30:43 on dist-test-slave-gqzk"
server_key: "d9b0d038d38ddbca27c2ef9d84485631"
server_key_iv: "e40449258e05a118d1c380d6c83559d0"
server_key_version: "encryptionkey@0"
I20260620 14:30:43.694895 7705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:43.709597 7705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:43.712710 7705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:43.712916 7705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:43.713492 7705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:43.714300 7705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:43.714354 7705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.714406 7705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:43.714429 7705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:43.722996 7705 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:41839
I20260620 14:30:43.723019 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:41839 every 8 connection(s)
I20260620 14:30:43.723872 7705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:43.731307 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7705
I20260620 14:30:43.731451 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:43.731758 4329 external_mini_cluster.cc:1428] Setting key f39afa12f9a7f1e00de8c5b7ae627c1b
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:43.738006 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.725643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46039 (local address 127.4.58.126:44541)
0620 14:30:43.725788 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:43.725791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:43.726631 (+ 840us) server_negotiation.cc:407] Connection header received
0620 14:30:43.727998 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:43.728001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:43.728053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:43.728133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:43.729477 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.730001 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.731204 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.731428 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.731530 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.734630 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:43.734638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:43.734641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:43.734669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:43.735987 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.736479 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.736482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.736484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.736563 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.736946 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.736959 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.736961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.737146 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:43.737233 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:43.737728 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:30:43.737850 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:30:43.738798 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.725994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44541 (local address 127.4.58.67:46039)
0620 14:30:43.726471 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:43.726514 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:43.727750 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:43.728381 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:43.728390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:43.728794 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:43.729304 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.729318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.730121 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.730124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.731060 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:43.731069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.731590 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.731595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:43.731793 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.732460 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:43.732480 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:43.734459 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:43.736107 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.736113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.736129 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.736382 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.736644 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:43.736647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:43.736649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:43.736860 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:43.737241 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:43.737247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:43.737493 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 14:30:43.737652 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:30:43.737983 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":100,"threads_started":1}
I20260620 14:30:43.739848 7834 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44541
I20260620 14:30:43.740074 7834 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:43.740538 7834 heartbeater.cc:511] Master 127.4.58.126:44541 requested a full tablet report, sending...
I20260620 14:30:43.741503 7375 ts_manager.cc:194] Registered new tserver with Master: 6debe7a0f8b246aba3ed081225eb0123 (127.4.58.67:41839)
I20260620 14:30:43.742038 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:46039
I20260620 14:30:43.745541 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965843, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:43.769449 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:43.761399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.4.58.126:44541)
0620 14:30:43.761551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:43.761556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:43.761657 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:30:43.761775 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:43.761777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:43.761824 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:43.761884 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:43.762520 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.763034 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.763874 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:43.764099 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:43.764179 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:43.766554 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:43.766558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:43.766560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:43.766586 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:43.767887 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.768274 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.768276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.768278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.768344 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:43.768562 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:43.768564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:43.768565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:43.768832 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:43.768928 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:43.769162 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:30:43.769260 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:30:43.769879 7375 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59380
I20260620 14:30:43.771236 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7437
I20260620 14:30:43.776083 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7571
I20260620 14:30:43.780848 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7705
I20260620 14:30:43.785532 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7343
2026-06-20T14:30:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2060 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7846](info): set up 2 sockets
Jun 20 14:30:43 dist-test-slave-gqzk krb5kdc[7846](info): commencing operation
krb5kdc: starting...
W20260620 14:30:44.829186 4329 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 14:30:45.106150 4329 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.277s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 14:30:45.343634 4329 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.237s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:30:45.554361 4329 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.211s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/test-user.keytab.
W20260620 14:30:46.462762 4329 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.908s user 0.002s sys 0.000s
Jun 20 14:30:46 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
W20260620 14:30:47.688196 4329 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.58.126: real 1.203s user 0.001s sys 0.002s
I20260620 14:30:47.688720 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:37271
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:47.797665 7859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:47.797940 7859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:47.797981 7859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:47.801961 7859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:47.802026 7859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:47.802043 7859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:47.802057 7859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:47.802071 7859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:47.807382 7859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:47.808383 7859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:47.809381 7859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:47.817677 7865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:47.817773 7867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:47.817677 7864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:47.817981 7859 server_base.cc:1034] running on GCE node
I20260620 14:30:47.818574 7859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:47.819701 7859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:47.820824 7859 hybrid_clock.cc:648] HybridClock initialized: now 1781965847820801 us; error 37 us; skew 500 ppm
Jun 20 14:30:47 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965847, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:47.828073 7859 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:47.829211 7859 webserver.cc:460] Webserver started at http://127.4.58.126:34129/ using document root <none> and password file <none>
I20260620 14:30:47.829907 7859 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:47.829988 7859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:47.830211 7859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:47.832470 7859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "862bfa51808f40ee9b2b705ffa563034"
format_stamp: "Formatted at 2026-06-20 14:30:47 on dist-test-slave-gqzk"
server_key: "bfe052d385ce2434d981bd81178492e2"
server_key_iv: "df9fd15192a5a20010c319fb14b4fc93"
server_key_version: "encryptionkey@0"
I20260620 14:30:47.833041 7859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "862bfa51808f40ee9b2b705ffa563034"
format_stamp: "Formatted at 2026-06-20 14:30:47 on dist-test-slave-gqzk"
server_key: "bfe052d385ce2434d981bd81178492e2"
server_key_iv: "df9fd15192a5a20010c319fb14b4fc93"
server_key_version: "encryptionkey@0"
I20260620 14:30:47.836984 7859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:30:47.839542 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:47.840937 7859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:47.841054 7859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "862bfa51808f40ee9b2b705ffa563034"
format_stamp: "Formatted at 2026-06-20 14:30:47 on dist-test-slave-gqzk"
server_key: "bfe052d385ce2434d981bd81178492e2"
server_key_iv: "df9fd15192a5a20010c319fb14b4fc93"
server_key_version: "encryptionkey@0"
I20260620 14:30:47.841146 7859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:47.868872 7859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:47.872020 7859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:47.872179 7859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:47.879266 7859 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:37271
I20260620 14:30:47.879280 7926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:37271 every 8 connection(s)
I20260620 14:30:47.880328 7859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:47.883168 7927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:47.885042 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7859
I20260620 14:30:47.885149 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:47.885421 4329 external_mini_cluster.cc:1428] Setting key 95ca78f9afe40e1ef3ab97ab3daeb8c8
I20260620 14:30:47.890252 7927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034: Bootstrap starting.
I20260620 14:30:47.893018 7927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:47.893815 7927 log.cc:824] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:30:47 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:47.896140 7927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034: No bootstrap required, opened a new log
I20260620 14:30:47.900887 7927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } }
I20260620 14:30:47.901167 7927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:47.901237 7927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862bfa51808f40ee9b2b705ffa563034, State: Initialized, Role: FOLLOWER
I20260620 14:30:47.901505 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:47.886945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.4.58.126:37271)
0620 14:30:47.887380 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:47.887410 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:47.887456 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:30:47.888200 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:47.888227 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:47.888691 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:47.889125 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:47.889796 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:47.891282 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:47.892233 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:47.892960 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:47.893126 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:47.895638 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:47.895644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:47.895663 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:47.895698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:47.897970 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:47.898548 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:47.898556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:47.898564 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:47.898726 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:47.899068 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:47.899072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:47.899074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:47.899619 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:47.899820 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:47.900115 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:30:47.900438 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260620 14:30:47.901731 7927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } }
I20260620 14:30:47.901895 7927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:47.901961 7927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:47.902057 7927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:47.931447 7927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } }
I20260620 14:30:47.932037 7927 leader_election.cc:302] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862bfa51808f40ee9b2b705ffa563034; no voters:
I20260620 14:30:47.932497 7927 leader_election.cc:288] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:47.932632 7932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:47.932945 7932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [term 1 LEADER]: Becoming Leader. State: Replica: 862bfa51808f40ee9b2b705ffa563034, State: Running, Role: LEADER
I20260620 14:30:47.933364 7932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } }
I20260620 14:30:47.934031 7927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:47.935197 7933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862bfa51808f40ee9b2b705ffa563034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } } }
I20260620 14:30:47.935209 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862bfa51808f40ee9b2b705ffa563034. Latest consensus state: current_term: 1 leader_uuid: "862bfa51808f40ee9b2b705ffa563034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862bfa51808f40ee9b2b705ffa563034" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 37271 } } }
I20260620 14:30:47.935370 7933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:47.935370 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:47.935914 7941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:47.938920 7941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:47.945652 7941 catalog_manager.cc:1347] Generated new cluster ID: ec313ff1f0034b45b6a1111d5bfe5cde
I20260620 14:30:47.945760 7941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:47.967338 7941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:47.968889 7941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:47.980296 7941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862bfa51808f40ee9b2b705ffa563034: Generated new TSK 0
I20260620 14:30:47.981149 7941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
W20260620 14:30:48.627835 4329 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.58.65: real 0.641s user 0.002s sys 0.002s
I20260620 14:30:48.628320 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:48.735692 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:48.735954 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:48.735997 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:48.740155 7953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:48.740234 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:48.740299 7953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:48.746023 7953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:48.747049 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:48.747990 7953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:48.757697 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:48.757697 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:48.757809 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:48.758381 7953 server_base.cc:1034] running on GCE node
I20260620 14:30:48.758812 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:48.759485 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:48.760659 7953 hybrid_clock.cc:648] HybridClock initialized: now 1781965848760639 us; error 36 us; skew 500 ppm
Jun 20 14:30:48 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:48.763396 7953 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:48.764294 7953 webserver.cc:460] Webserver started at http://127.4.58.65:34093/ using document root <none> and password file <none>
I20260620 14:30:48.764906 7953 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:48.764981 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:48.765182 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:48.767247 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "08d3934ebdf648669633bfa82b0d78ae"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "20e1cdccdc86d60e2b545c5dc283be53"
server_key_iv: "9e8558273f46d29c6f48a303caaba13f"
server_key_version: "encryptionkey@0"
I20260620 14:30:48.767776 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "08d3934ebdf648669633bfa82b0d78ae"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "20e1cdccdc86d60e2b545c5dc283be53"
server_key_iv: "9e8558273f46d29c6f48a303caaba13f"
server_key_version: "encryptionkey@0"
I20260620 14:30:48.771600 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:30:48.774286 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:48.775763 7953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:30:48.775888 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "08d3934ebdf648669633bfa82b0d78ae"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "20e1cdccdc86d60e2b545c5dc283be53"
server_key_iv: "9e8558273f46d29c6f48a303caaba13f"
server_key_version: "encryptionkey@0"
I20260620 14:30:48.775998 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:48.806540 7953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:48.809890 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:48.810066 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:48.810670 7953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:48.811517 7953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:48.811568 7953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:48.811621 7953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:48.811647 7953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:48.820221 7953 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:34377
I20260620 14:30:48.820418 8081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:34377 every 8 connection(s)
I20260620 14:30:48.821214 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:48.824790 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 7953
I20260620 14:30:48.824942 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:48.825207 4329 external_mini_cluster.cc:1428] Setting key 0acbe7e6f6acfc24017e7677e8a99479
Jun 20 14:30:48 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:48.835126 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:48.823025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:35641 (local address 127.4.58.126:37271)
0620 14:30:48.823269 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:48.823274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:48.824046 (+ 772us) server_negotiation.cc:407] Connection header received
0620 14:30:48.825569 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:48.825573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:48.825636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:48.825720 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:48.826944 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:48.827474 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:48.828645 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:48.828920 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:48.829017 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:48.831460 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:48.831464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:48.831467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:48.831493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:48.832983 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:48.833522 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:48.833525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:48.833527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:48.833608 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:48.833939 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:48.833943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:48.833945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:48.834162 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:48.834252 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:48.834790 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 14:30:48.834930 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:48.836117 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:48.823443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:37271 (local address 127.4.58.65:35641)
0620 14:30:48.823892 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:48.823938 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:48.825328 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:48.825915 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:48.825922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:48.826286 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:48.826786 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:48.826801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:48.827598 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:48.827601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:48.828520 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:48.828530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:48.829069 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:48.829073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:48.829242 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:48.829828 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:48.829851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:48.831283 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:48.833098 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:48.833105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:48.833122 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:48.833425 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:48.833696 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:48.833699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:48.833701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:48.833847 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:48.834262 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:48.834269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:48.834544 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 14:30:48.834736 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:30:48.835093 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260620 14:30:48.837469 8082 heartbeater.cc:347] Connected to a master server at 127.4.58.126:37271
I20260620 14:30:48.837713 8082 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:48.838214 8082 heartbeater.cc:511] Master 127.4.58.126:37271 requested a full tablet report, sending...
I20260620 14:30:48.839951 7891 ts_manager.cc:194] Registered new tserver with Master: 08d3934ebdf648669633bfa82b0d78ae (127.4.58.65:34377)
I20260620 14:30:48.841693 7891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:35641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:48.851107 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:48.962419 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:48.962683 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:48.962724 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:48.967141 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:48.967213 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:48.967293 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:48.974808 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:48.976368 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:48.978045 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:48.989202 8093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:48.989223 8094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:48.989202 8096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:48.989975 8088 server_base.cc:1034] running on GCE node
I20260620 14:30:48.990499 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:48.991267 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:48.992450 8088 hybrid_clock.cc:648] HybridClock initialized: now 1781965848992410 us; error 60 us; skew 500 ppm
Jun 20 14:30:48 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:48.995540 8088 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:48.996591 8088 webserver.cc:460] Webserver started at http://127.4.58.66:36675/ using document root <none> and password file <none>
I20260620 14:30:48.997237 8088 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:48.997308 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:48.997478 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:48.999539 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "8f76fa5b9aed44af8e699866d2f12ab6"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "db829cc603d2c64e829fb3bff2943f24"
server_key_iv: "d629bb45508c6c2791ced4b94e1a8779"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.000048 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "8f76fa5b9aed44af8e699866d2f12ab6"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "db829cc603d2c64e829fb3bff2943f24"
server_key_iv: "d629bb45508c6c2791ced4b94e1a8779"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.003841 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:30:49.006448 8103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.007822 8088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:30:49.007934 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "8f76fa5b9aed44af8e699866d2f12ab6"
format_stamp: "Formatted at 2026-06-20 14:30:48 on dist-test-slave-gqzk"
server_key: "db829cc603d2c64e829fb3bff2943f24"
server_key_iv: "d629bb45508c6c2791ced4b94e1a8779"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.008037 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:49.019766 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:49.022974 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:49.023130 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:49.023679 8088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:49.024480 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:49.024534 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.024573 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:49.024593 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.032930 8088 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:38931
I20260620 14:30:49.032966 8216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:38931 every 8 connection(s)
I20260620 14:30:49.033865 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:49.037633 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8088
I20260620 14:30:49.037743 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:49.038007 4329 external_mini_cluster.cc:1428] Setting key f1a8b6ec29f8ec64a8b59995d8be150e
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:49.048336 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.035684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:54585 (local address 127.4.58.126:37271)
0620 14:30:49.035813 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:49.035817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:49.036734 (+ 917us) server_negotiation.cc:407] Connection header received
0620 14:30:49.038202 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:49.038205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:49.038266 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:49.038342 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:49.040010 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.040539 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.041769 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.042005 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.042095 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.044821 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:49.044824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:49.044826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:49.044855 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:49.046282 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.046906 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.046909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.046910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.046992 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.047296 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.047298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.047299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.047485 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:49.047557 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:49.048079 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:30:49.048206 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:30:49.049290 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.036077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:37271 (local address 127.4.58.66:54585)
0620 14:30:49.036588 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:49.036631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:49.037971 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:49.038721 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:49.038731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:49.039227 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:49.039855 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:49.039873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.040664 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.040667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:49.041640 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:49.041650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.042149 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.042153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:49.042320 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.042921 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:49.042938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:49.044605 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:49.046411 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:49.046419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:49.046439 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:49.046787 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:49.047075 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:49.047078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:49.047079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:49.047226 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:49.047564 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:49.047569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:49.047848 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 14:30:49.048023 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:30:49.048316 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":111,"threads_started":1}
I20260620 14:30:49.050612 8217 heartbeater.cc:347] Connected to a master server at 127.4.58.126:37271
I20260620 14:30:49.050889 8217 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:49.051492 8217 heartbeater.cc:511] Master 127.4.58.126:37271 requested a full tablet report, sending...
I20260620 14:30:49.052582 7891 ts_manager.cc:194] Registered new tserver with Master: 8f76fa5b9aed44af8e699866d2f12ab6 (127.4.58.66:38931)
I20260620 14:30:49.053231 7891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:54585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:49.063644 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:49.182053 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:49.182340 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:49.182402 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:49.186641 8222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:49.186709 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:49.186790 8222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:49.192224 8222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:37271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:49.193394 8222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:49.194422 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:49.203956 8228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:49.204069 8222 server_base.cc:1034] running on GCE node
W20260620 14:30:49.203964 8227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:49.203979 8230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:49.204674 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:49.205372 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:49.206537 8222 hybrid_clock.cc:648] HybridClock initialized: now 1781965849206517 us; error 32 us; skew 500 ppm
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965849, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:49.209254 8222 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:49.210167 8222 webserver.cc:460] Webserver started at http://127.4.58.67:41489/ using document root <none> and password file <none>
I20260620 14:30:49.210702 8222 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:49.210767 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:49.210994 8222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:49.213176 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "46ba95f460b741a595c895c73b810fd1"
format_stamp: "Formatted at 2026-06-20 14:30:49 on dist-test-slave-gqzk"
server_key: "4d74ea1c425c0448b1e9eb52879b607c"
server_key_iv: "2b4e2d7325e0126ee4cbcdf3994d0a23"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.213723 8222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "46ba95f460b741a595c895c73b810fd1"
format_stamp: "Formatted at 2026-06-20 14:30:49 on dist-test-slave-gqzk"
server_key: "4d74ea1c425c0448b1e9eb52879b607c"
server_key_iv: "2b4e2d7325e0126ee4cbcdf3994d0a23"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.217386 8222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:49.219723 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.221144 8222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:49.221271 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "46ba95f460b741a595c895c73b810fd1"
format_stamp: "Formatted at 2026-06-20 14:30:49 on dist-test-slave-gqzk"
server_key: "4d74ea1c425c0448b1e9eb52879b607c"
server_key_iv: "2b4e2d7325e0126ee4cbcdf3994d0a23"
server_key_version: "encryptionkey@0"
I20260620 14:30:49.221382 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:49.237401 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:49.241206 8222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:49.241461 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:49.242144 8222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:49.243041 8222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:49.243088 8222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.243142 8222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:49.243167 8222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:49.251703 8222 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:43209
I20260620 14:30:49.251734 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:43209 every 8 connection(s)
I20260620 14:30:49.252610 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:49.260051 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8222
I20260620 14:30:49.260188 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:49.260524 4329 external_mini_cluster.cc:1428] Setting key 675ec03668762e629bc3c178adb14a56
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965849, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:49.266026 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.254459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:34951 (local address 127.4.58.126:37271)
0620 14:30:49.254592 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:49.254596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:49.255472 (+ 876us) server_negotiation.cc:407] Connection header received
0620 14:30:49.256743 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:49.256745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:49.256833 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:49.256906 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:49.258071 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.258622 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.259718 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.259934 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.260007 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.262819 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:49.262822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:49.262824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:49.262848 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:49.264092 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.264564 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.264567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.264568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.264639 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.264977 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.264980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.264982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.265161 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:49.265245 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:49.265736 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:30:49.265868 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:30:49.266875 8353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.254828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:37271 (local address 127.4.58.67:34951)
0620 14:30:49.255282 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:49.255329 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:49.256564 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:49.257074 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:49.257081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:49.257463 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:49.257921 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:49.257934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.258738 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.258741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:49.259624 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:49.259633 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.260076 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.260081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:49.260276 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.260987 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:49.261008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:49.262662 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:49.264202 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:49.264208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:49.264225 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:49.264473 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:49.264716 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:49.264719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:49.264721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:49.264893 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:49.265255 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:49.265262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:49.265519 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 14:30:49.265685 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:30:49.266015 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260620 14:30:49.268070 8351 heartbeater.cc:347] Connected to a master server at 127.4.58.126:37271
I20260620 14:30:49.268285 8351 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:49.268779 8351 heartbeater.cc:511] Master 127.4.58.126:37271 requested a full tablet report, sending...
I20260620 14:30:49.269835 7891 ts_manager.cc:194] Registered new tserver with Master: 46ba95f460b741a595c895c73b810fd1 (127.4.58.67:43209)
I20260620 14:30:49.270460 7891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:34951
I20260620 14:30:49.274528 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:49.299477 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.289975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.4.58.65:34377)
0620 14:30:49.290270 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:49.290286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:49.290314 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:30:49.290422 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:49.290427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:49.290629 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:49.290767 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:49.291500 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.292273 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.293194 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.293601 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.293701 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.296011 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:49.296021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:49.296036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:49.296063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:49.297593 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.298013 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.298018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.298020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.298106 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.298461 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.298464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.298465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.298724 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:49.298943 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:49.299141 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:30:49.299295 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[7846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:49.310019 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:49.302265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.4.58.126:37271)
0620 14:30:49.302393 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:49.302398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:49.302483 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:30:49.302602 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:49.302605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:49.302652 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:49.302714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:49.303278 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.303798 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.304556 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:49.304827 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:49.304918 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:49.307247 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:49.307251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:49.307254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:49.307284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:49.308401 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.308820 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.308824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.308826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.308922 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:49.309161 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:49.309164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:49.309166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:49.309401 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:49.309522 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:49.309740 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:30:49.309854 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 14:30:49.310402 7891 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33252
I20260620 14:30:49.310997 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7953
I20260620 14:30:49.316478 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8088
I20260620 14:30:49.321332 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8222
I20260620 14:30:49.325896 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 7859
2026-06-20T14:30:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5540 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[8363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[8363](info): set up 2 sockets
Jun 20 14:30:49 dist-test-slave-gqzk krb5kdc[8363](info): commencing operation
krb5kdc: starting...
W20260620 14:30:50.276701 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:50 dist-test-slave-gqzk krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:50.392450 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34761
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:50.502739 8376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:50.503041 8376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:50.503105 8376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:50.507253 8376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:50.507335 8376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:50.507362 8376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:50.507382 8376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:50.507404 8376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:50.512835 8376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:50.513957 8376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:50.515000 8376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:50.523377 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:50.523382 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:50.523418 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:50.523888 8376 server_base.cc:1034] running on GCE node
I20260620 14:30:50.524595 8376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:50.526050 8376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:50.527230 8376 hybrid_clock.cc:648] HybridClock initialized: now 1781965850527209 us; error 32 us; skew 500 ppm
Jun 20 14:30:50 dist-test-slave-gqzk krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:50.530119 8376 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:50.530920 8376 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:50.530987 8376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:50.531153 8376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:50.533315 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "cf5f4287e92a414eab058a4af5a22933"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "e83177515c184731442c9563f9ea94e4"
server_key_iv: "4b1f2154f9c85b3ce7642fb52aa099af"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.533864 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "cf5f4287e92a414eab058a4af5a22933"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "e83177515c184731442c9563f9ea94e4"
server_key_iv: "4b1f2154f9c85b3ce7642fb52aa099af"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.537649 8376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:50.540169 8390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:50.541548 8376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:50.541661 8376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "cf5f4287e92a414eab058a4af5a22933"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "e83177515c184731442c9563f9ea94e4"
server_key_iv: "4b1f2154f9c85b3ce7642fb52aa099af"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.541757 8376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:50.561156 8376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:50.564170 8376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:50.564330 8376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:50.571276 8376 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34761
I20260620 14:30:50.571532 8442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34761 every 8 connection(s)
I20260620 14:30:50.571966 8376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:50.574882 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:50.578739 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8376
I20260620 14:30:50.578894 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:50.579180 4329 external_mini_cluster.cc:1428] Setting key c21b5d7b76326d1b6e06bf49d3c0bece
I20260620 14:30:50.580971 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: Bootstrap starting.
I20260620 14:30:50.583148 8443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:50.583858 8443 log.cc:824] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:50.585681 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: No bootstrap required, opened a new log
Jun 20 14:30:50 dist-test-slave-gqzk krb5kdc[8363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:50.589665 8443 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } }
I20260620 14:30:50.589898 8443 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:50.589932 8443 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf5f4287e92a414eab058a4af5a22933, State: Initialized, Role: FOLLOWER
I20260620 14:30:50.590404 8443 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } }
I20260620 14:30:50.590543 8443 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:50.590637 8443 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:50.590734 8443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:50.591591 8443 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } }
I20260620 14:30:50.591928 8443 leader_election.cc:302] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf5f4287e92a414eab058a4af5a22933; no voters:
I20260620 14:30:50.592265 8443 leader_election.cc:288] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:50.592348 8448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:50.592530 8448 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [term 1 LEADER]: Becoming Leader. State: Replica: cf5f4287e92a414eab058a4af5a22933, State: Running, Role: LEADER
I20260620 14:30:50.593225 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:50.580632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.4.58.126:34761)
0620 14:30:50.581080 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:50.581101 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:50.581140 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:30:50.581767 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:50.581787 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:50.582134 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:50.582452 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:50.583197 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.584233 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.585118 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.585582 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.585722 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:50.588235 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:50.588240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:50.588254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:50.588280 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:50.590098 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:50.590519 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:50.590525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:50.590531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:50.590688 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:50.590972 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:50.590975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:50.590977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:50.591379 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:50.591565 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:50.591858 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:30:50.592160 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":158,"threads_started":1}
I20260620 14:30:50.593516 8443 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:50.594501 8448 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } }
I20260620 14:30:50.596135 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf5f4287e92a414eab058a4af5a22933. Latest consensus state: current_term: 1 leader_uuid: "cf5f4287e92a414eab058a4af5a22933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } } }
I20260620 14:30:50.596261 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:50.596575 8449 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf5f4287e92a414eab058a4af5a22933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf5f4287e92a414eab058a4af5a22933" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34761 } } }
I20260620 14:30:50.596720 8462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:50.596743 8449 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:50.600234 8462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:50.605994 8462 catalog_manager.cc:1347] Generated new cluster ID: bacd6bdf71ac41f0aefb3ecd5172669f
I20260620 14:30:50.606058 8462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:50.636263 8462 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:50.637315 8462 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:50.643484 8462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: Generated new TSK 0
I20260620 14:30:50.644167 8462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:50.675448 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:50.783339 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:50.783608 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:50.783670 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:50.787662 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:50.787730 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:50.787814 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:50.793115 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:50.794212 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:50.795266 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:50.804593 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:50.804589 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:50.804706 8469 server_base.cc:1034] running on GCE node
W20260620 14:30:50.804646 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:50.805308 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:50.805915 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:50.807071 8469 hybrid_clock.cc:648] HybridClock initialized: now 1781965850807050 us; error 37 us; skew 500 ppm
Jun 20 14:30:50 dist-test-slave-gqzk krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:50.809754 8469 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:50.810436 8469 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:50.810495 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:50.810676 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:50.812670 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "56b5d3e91ea449848d118842a30ccedb"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "13a7b7a183a1a4f60e616f0b593a9c38"
server_key_iv: "e1f0e4bd4cae524fd14cf51f92dec8a5"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.813220 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "56b5d3e91ea449848d118842a30ccedb"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "13a7b7a183a1a4f60e616f0b593a9c38"
server_key_iv: "e1f0e4bd4cae524fd14cf51f92dec8a5"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.816726 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:30:50.819125 8483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:50.820384 8469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:50.820499 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "56b5d3e91ea449848d118842a30ccedb"
format_stamp: "Formatted at 2026-06-20 14:30:50 on dist-test-slave-gqzk"
server_key: "13a7b7a183a1a4f60e616f0b593a9c38"
server_key_iv: "e1f0e4bd4cae524fd14cf51f92dec8a5"
server_key_version: "encryptionkey@0"
I20260620 14:30:50.820600 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:50.846817 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:50.850072 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:50.850247 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:50.850834 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:50.851673 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:50.851727 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:50.851778 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:50.851802 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:50.859993 8469 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:39819
I20260620 14:30:50.860033 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:39819 every 8 connection(s)
I20260620 14:30:50.860569 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:50.861989 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8469
I20260620 14:30:50.862111 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:50.862491 4329 external_mini_cluster.cc:1428] Setting key 398d9d8ba98b8edc244b45217310b612
Jun 20 14:30:50 dist-test-slave-gqzk krb5kdc[8363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:50.875624 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:50.862410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:51333 (local address 127.4.58.126:34761)
0620 14:30:50.862562 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:50.862567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:50.864122 (+ 1555us) server_negotiation.cc:407] Connection header received
0620 14:30:50.865557 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:50.865560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:50.865608 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:50.865667 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:50.867195 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.867771 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.869153 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.869392 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.869483 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:50.871876 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:50.871879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:50.871881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:50.871907 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:50.873274 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:50.873930 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:50.873933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:50.873935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:50.874030 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:50.874359 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:50.874363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:50.874365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:50.874642 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:50.874736 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:50.875267 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 14:30:50.875453 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:30:50.876582 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:50.862953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34761 (local address 127.4.58.65:51333)
0620 14:30:50.863959 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:50.864020 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:50.865295 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:50.865861 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:50.865870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:50.866381 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:50.867039 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:50.867058 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.867918 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.867922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:50.869033 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:50.869043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:50.869539 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:50.869543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:50.869707 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:50.870245 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:50.870263 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:50.871671 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:50.873446 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:50.873452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:50.873469 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:50.873822 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:50.874124 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:50.874127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:50.874128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:50.874274 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:50.874740 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:50.874745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:50.875026 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 14:30:50.875198 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:30:50.875577 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":148,"threads_started":1}
I20260620 14:30:50.877977 8597 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34761
I20260620 14:30:50.878257 8597 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:50.878751 8597 heartbeater.cc:511] Master 127.4.58.126:34761 requested a full tablet report, sending...
I20260620 14:30:50.880326 8407 ts_manager.cc:194] Registered new tserver with Master: 56b5d3e91ea449848d118842a30ccedb (127.4.58.65:39819)
I20260620 14:30:50.881516 8407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:51333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:50.889775 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:51.000329 8602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:51.000607 8602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:51.000670 8602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:51.004784 8602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:51.004880 8602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:51.004978 8602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:51.010499 8602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:51.011628 8602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:51.012792 8602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:51.022217 8610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.022213 8608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.022212 8607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:51.022478 8602 server_base.cc:1034] running on GCE node
I20260620 14:30:51.022969 8602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:51.023619 8602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:51.024801 8602 hybrid_clock.cc:648] HybridClock initialized: now 1781965851024765 us; error 44 us; skew 500 ppm
Jun 20 14:30:51 dist-test-slave-gqzk krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:51.027601 8602 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:51.028321 8602 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:51.028393 8602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:51.028590 8602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:51.030752 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "72caaa20959a43338ed1fb11bfac7520"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "36e32d4b7c274e1eebbd3c435264384b"
server_key_iv: "846ef9454667cf77b959bf67ddf2c903"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.031327 8602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "72caaa20959a43338ed1fb11bfac7520"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "36e32d4b7c274e1eebbd3c435264384b"
server_key_iv: "846ef9454667cf77b959bf67ddf2c903"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.035074 8602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:51.037762 8616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.039065 8602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:51.039187 8602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "72caaa20959a43338ed1fb11bfac7520"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "36e32d4b7c274e1eebbd3c435264384b"
server_key_iv: "846ef9454667cf77b959bf67ddf2c903"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.039291 8602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:51.058719 8602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:51.061962 8602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:51.062145 8602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:51.062747 8602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:51.063576 8602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:51.063623 8602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.063676 8602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:51.063705 8602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.072280 8602 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46643
I20260620 14:30:51.072306 8729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46643 every 8 connection(s)
I20260620 14:30:51.072932 8602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:51.076195 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8602
I20260620 14:30:51.076318 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:51.076594 4329 external_mini_cluster.cc:1428] Setting key 1cc90761560d6434c1971669784e1261
Jun 20 14:30:51 dist-test-slave-gqzk krb5kdc[8363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:51.087387 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.074729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:40509 (local address 127.4.58.126:34761)
0620 14:30:51.074886 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.074889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.075762 (+ 873us) server_negotiation.cc:407] Connection header received
0620 14:30:51.077187 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.077190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.077238 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.077313 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:51.078544 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.079066 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.080668 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.080922 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.081004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.083723 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:51.083726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:51.083728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:51.083754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:51.085207 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.085720 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.085725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.085727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.085834 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.086123 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.086125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.086126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.086310 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:51.086402 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.087090 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.087203 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:30:51.088390 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.075111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34761 (local address 127.4.58.66:40509)
0620 14:30:51.075598 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.075643 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.076960 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.077492 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.077499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.077921 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:51.078410 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.078423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.079183 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.079187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.080540 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.080551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.081070 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.081074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.081260 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.081899 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:51.081922 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:51.083543 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:51.085318 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:51.085325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:51.085339 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:51.085589 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:51.085917 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:51.085919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:51.085921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:51.086052 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:51.086432 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:51.086438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:51.086823 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.087001 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.087322 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260620 14:30:51.089600 8730 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34761
I20260620 14:30:51.089833 8730 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:51.090271 8730 heartbeater.cc:511] Master 127.4.58.126:34761 requested a full tablet report, sending...
I20260620 14:30:51.091331 8407 ts_manager.cc:194] Registered new tserver with Master: 72caaa20959a43338ed1fb11bfac7520 (127.4.58.66:46643)
I20260620 14:30:51.091908 8407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:40509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:51.101843 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:51.211436 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:51.211689 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:51.211733 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:51.215713 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:51.215776 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:51.215837 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:51.221287 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:51.222281 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:51.223223 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:51.232281 8735 server_base.cc:1034] running on GCE node
W20260620 14:30:51.232265 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.232260 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.232489 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:51.232996 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:51.233628 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:51.234781 8735 hybrid_clock.cc:648] HybridClock initialized: now 1781965851234753 us; error 39 us; skew 500 ppm
Jun 20 14:30:51 dist-test-slave-gqzk krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:51.237581 8735 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:51.238268 8735 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:51.238329 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:51.238512 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:51.240483 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "a39e78f386b749f891e9bab9fd8e9f9f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "c4dd52f9f5668029a5b7ce638f8bfe16"
server_key_iv: "26353a847efba6723ec13e8a06a560e7"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.241029 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "a39e78f386b749f891e9bab9fd8e9f9f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "c4dd52f9f5668029a5b7ce638f8bfe16"
server_key_iv: "26353a847efba6723ec13e8a06a560e7"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.244658 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:51.247051 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.248353 8735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:51.248489 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "a39e78f386b749f891e9bab9fd8e9f9f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "c4dd52f9f5668029a5b7ce638f8bfe16"
server_key_iv: "26353a847efba6723ec13e8a06a560e7"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.248581 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:51.274983 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:51.278241 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:51.278419 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:51.279006 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:51.279834 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:51.279881 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.279932 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:51.279960 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.288558 8735 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:33563
I20260620 14:30:51.288584 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:33563 every 8 connection(s)
I20260620 14:30:51.289165 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:51.298152 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8735
I20260620 14:30:51.298257 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
Jun 20 14:30:51 dist-test-slave-gqzk krb5kdc[8363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965851, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:51.298519 4329 external_mini_cluster.cc:1428] Setting key eef778d3df4caa038f9de449a5a1d43c
I20260620 14:30:51.302661 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.290840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:34293 (local address 127.4.58.126:34761)
0620 14:30:51.291010 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.291015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.291871 (+ 856us) server_negotiation.cc:407] Connection header received
0620 14:30:51.293224 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.293227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.293275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.293339 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:51.294516 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.295087 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.296252 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.296495 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.296571 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.298874 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:51.298877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:51.298880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:51.298903 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:51.300495 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.301104 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.301107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.301109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.301199 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.301540 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.301543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.301545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.301729 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:51.301814 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.302336 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.302502 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:30:51.303436 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.291282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34761 (local address 127.4.58.67:34293)
0620 14:30:51.291718 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.291761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.292990 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.293498 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.293505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.293850 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:51.294397 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.294411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.295180 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.295183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.296160 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.296169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.296622 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.296625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.296785 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.297301 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:51.297316 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:51.298721 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:51.300627 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:51.300634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:51.300651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:51.301009 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:51.301292 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:51.301295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:51.301297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:51.301445 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:51.301826 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:51.301833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:51.302099 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.302260 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.302606 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260620 14:30:51.304481 8863 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34761
I20260620 14:30:51.304693 8863 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:51.305142 8863 heartbeater.cc:511] Master 127.4.58.126:34761 requested a full tablet report, sending...
I20260620 14:30:51.306098 8407 ts_manager.cc:194] Registered new tserver with Master: a39e78f386b749f891e9bab9fd8e9f9f (127.4.58.67:33563)
I20260620 14:30:51.306628 8407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:34293
I20260620 14:30:51.312501 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:51.321201 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.314376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.4.58.126:34761)
0620 14:30:51.314523 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.314526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.314632 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:30:51.314732 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.314735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.314772 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.314838 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:51.315459 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.315969 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.316822 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.317059 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.317139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.318390 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:51.318393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:51.318395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:51.318418 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:51.319657 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.320025 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.320028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.320029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.320098 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:51.320367 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:51.320370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:51.320372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:51.320645 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:51.320804 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.320987 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.321086 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:30:51.324052 8407 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:51.327085 8407 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:51.341729 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.335391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46643 (local address 127.0.0.1:36512)
0620 14:30:51.336540 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.336574 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.336673 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.337602 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.337605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.337623 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:51.337726 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.337736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.339098 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.339101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.339949 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.339959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.341018 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.341022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.341199 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341277 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.341393 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.341549 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":75,"thread_start_us":81,"threads_started":1}
I20260620 14:30:51.341784 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.336206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.4.58.67:33563)
0620 14:30:51.336590 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.336602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.336948 (+ 346us) server_negotiation.cc:407] Connection header received
0620 14:30:51.337049 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.337055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.337255 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.337402 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:51.337832 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.338608 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.340281 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.341071 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.341163 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341291 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.341513 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.341617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":111,"threads_started":1}
I20260620 14:30:51.341809 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.336185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.4.58.65:39819)
0620 14:30:51.336932 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.336939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.336961 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:30:51.337049 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.337054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.337232 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.337383 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:51.337662 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.338893 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.340392 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.341000 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.341358 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341508 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.341551 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.341674 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":105,"threads_started":1}
I20260620 14:30:51.341729 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.335462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:39819 (local address 127.0.0.1:56290)
0620 14:30:51.336521 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.336573 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.336675 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.337374 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.337380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.337401 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:51.337535 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.337542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.339044 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.339048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.340290 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.340299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.341120 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.341124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.341277 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341385 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.341436 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.341561 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":234,"thread_start_us":109,"threads_started":1}
I20260620 14:30:51.342162 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.335506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.4.58.66:46643)
0620 14:30:51.337171 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.337181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.337200 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:30:51.337276 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.337281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.337441 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.337608 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:51.337853 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.338978 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.340077 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.340884 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.340989 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341144 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.341824 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.342011 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":133,"threads_started":1}
I20260620 14:30:51.342514 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.335736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:33563 (local address 127.0.0.1:60894)
0620 14:30:51.336727 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.336739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.336862 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.337550 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.337554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.337571 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:51.337714 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.337721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.338752 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.338755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.340152 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.340163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.341193 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.341196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.341273 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.341348 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.341412 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.342330 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":88,"thread_start_us":84,"threads_started":1}
I20260620 14:30:51.345096 8664 tablet_service.cc:1467] Processing CreateTablet for tablet 650c2f387a574aceae155a5eac425975 (DEFAULT_TABLE table=test-table [id=92a1bdcd8fbe458ab220d6bd9ce04ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:51.345108 8797 tablet_service.cc:1467] Processing CreateTablet for tablet 650c2f387a574aceae155a5eac425975 (DEFAULT_TABLE table=test-table [id=92a1bdcd8fbe458ab220d6bd9ce04ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:51.345108 8531 tablet_service.cc:1467] Processing CreateTablet for tablet 650c2f387a574aceae155a5eac425975 (DEFAULT_TABLE table=test-table [id=92a1bdcd8fbe458ab220d6bd9ce04ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:51.346134 8664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c2f387a574aceae155a5eac425975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:51.346150 8797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c2f387a574aceae155a5eac425975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:51.346143 8531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c2f387a574aceae155a5eac425975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:51.353078 8880 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Bootstrap starting.
I20260620 14:30:51.354024 8882 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: Bootstrap starting.
I20260620 14:30:51.354024 8881 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Bootstrap starting.
I20260620 14:30:51.355201 8880 tablet_bootstrap.cc:654] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:51.356184 8881 tablet_bootstrap.cc:654] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:51.356184 8882 tablet_bootstrap.cc:654] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:51.356282 8880 log.cc:824] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:51.357507 8882 log.cc:824] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:51.357507 8881 log.cc:824] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:51.358227 8880 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: No bootstrap required, opened a new log
I20260620 14:30:51.358421 8880 ts_tablet_manager.cc:1397] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:30:51.359449 8882 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: No bootstrap required, opened a new log
I20260620 14:30:51.359449 8881 tablet_bootstrap.cc:492] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: No bootstrap required, opened a new log
I20260620 14:30:51.359632 8882 ts_tablet_manager.cc:1397] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:30:51.359632 8881 ts_tablet_manager.cc:1397] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:30:51.361491 8880 raft_consensus.cc:348] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.361763 8880 raft_consensus.cc:374] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:51.361822 8880 raft_consensus.cc:729] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72caaa20959a43338ed1fb11bfac7520, State: Initialized, Role: FOLLOWER
I20260620 14:30:51.362363 8880 consensus_queue.cc:260] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.363551 8880 ts_tablet_manager.cc:1428] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:30:51.363415 8882 raft_consensus.cc:348] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.363603 8730 heartbeater.cc:503] Master 127.4.58.126:34761 was elected leader, sending a full tablet report...
I20260620 14:30:51.363670 8882 raft_consensus.cc:374] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:51.363725 8882 raft_consensus.cc:729] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b5d3e91ea449848d118842a30ccedb, State: Initialized, Role: FOLLOWER
I20260620 14:30:51.364159 8882 consensus_queue.cc:260] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.364828 8597 heartbeater.cc:503] Master 127.4.58.126:34761 was elected leader, sending a full tablet report...
I20260620 14:30:51.365171 8882 ts_tablet_manager.cc:1428] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:30:51.363415 8881 raft_consensus.cc:348] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.365834 8881 raft_consensus.cc:374] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:51.365911 8881 raft_consensus.cc:729] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39e78f386b749f891e9bab9fd8e9f9f, State: Initialized, Role: FOLLOWER
I20260620 14:30:51.366308 8881 consensus_queue.cc:260] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.366940 8863 heartbeater.cc:503] Master 127.4.58.126:34761 was elected leader, sending a full tablet report...
I20260620 14:30:51.367421 8881 ts_tablet_manager.cc:1428] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:30:51.500871 8886 raft_consensus.cc:482] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:51.501094 8886 raft_consensus.cc:504] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.502295 8886 leader_election.cc:288] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56b5d3e91ea449848d118842a30ccedb (127.4.58.65:39819), a39e78f386b749f891e9bab9fd8e9f9f (127.4.58.67:33563)
I20260620 14:30:51.507148 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.502764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34137 (local address 127.4.58.67:33563)
0620 14:30:51.502915 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.502920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.503281 (+ 361us) server_negotiation.cc:407] Connection header received
0620 14:30:51.503355 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.503357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.503398 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.503471 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:51.503677 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.504397 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.505907 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.506499 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.506589 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.506647 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.506943 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.507024 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:30:51.507148 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.502703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:44139 (local address 127.4.58.65:39819)
0620 14:30:51.502866 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.502871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.502891 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:30:51.503020 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.503024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.503083 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.503170 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:51.503462 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.504215 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.505839 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.506459 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.506579 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.506647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.506943 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.507030 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:30:51.507138 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.502624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:39819 (local address 127.4.58.66:44139)
0620 14:30:51.502750 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.502768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.502883 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.503199 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.503202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.503225 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:51.503346 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.503352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.504353 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.504356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.505693 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.505704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.506602 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.506606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.506744 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.506790 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.506880 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.506973 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":11}
I20260620 14:30:51.507145 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.502624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:33563 (local address 127.4.58.66:34137)
0620 14:30:51.503183 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.503195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:51.503280 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:51.503480 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:51.503483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:51.503500 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:51.503584 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.503588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.504500 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.504503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.505780 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:51.505788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.506602 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.506606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:51.506769 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.506804 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:30:51.506877 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:30:51.506973 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":280,"thread_start_us":146,"threads_started":1}
I20260620 14:30:51.507949 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c2f387a574aceae155a5eac425975" candidate_uuid: "72caaa20959a43338ed1fb11bfac7520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b5d3e91ea449848d118842a30ccedb" is_pre_election: true
I20260620 14:30:51.507975 8817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c2f387a574aceae155a5eac425975" candidate_uuid: "72caaa20959a43338ed1fb11bfac7520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" is_pre_election: true
I20260620 14:30:51.508235 8817 raft_consensus.cc:2454] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72caaa20959a43338ed1fb11bfac7520 in term 0.
I20260620 14:30:51.508245 8551 raft_consensus.cc:2454] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72caaa20959a43338ed1fb11bfac7520 in term 0.
I20260620 14:30:51.508771 8620 leader_election.cc:302] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56b5d3e91ea449848d118842a30ccedb, 72caaa20959a43338ed1fb11bfac7520, a39e78f386b749f891e9bab9fd8e9f9f; no voters:
I20260620 14:30:51.509023 8886 raft_consensus.cc:2781] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:51.509099 8886 raft_consensus.cc:482] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:51.509151 8886 raft_consensus.cc:3037] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:51.510169 8886 raft_consensus.cc:504] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.510507 8886 leader_election.cc:288] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [CANDIDATE]: Term 1 election: Requested vote from peers 56b5d3e91ea449848d118842a30ccedb (127.4.58.65:39819), a39e78f386b749f891e9bab9fd8e9f9f (127.4.58.67:33563)
I20260620 14:30:51.510857 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c2f387a574aceae155a5eac425975" candidate_uuid: "72caaa20959a43338ed1fb11bfac7520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b5d3e91ea449848d118842a30ccedb"
I20260620 14:30:51.510885 8817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c2f387a574aceae155a5eac425975" candidate_uuid: "72caaa20959a43338ed1fb11bfac7520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39e78f386b749f891e9bab9fd8e9f9f"
I20260620 14:30:51.510998 8817 raft_consensus.cc:3037] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:51.510986 8551 raft_consensus.cc:3037] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:51.512001 8551 raft_consensus.cc:2454] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72caaa20959a43338ed1fb11bfac7520 in term 1.
I20260620 14:30:51.512020 8817 raft_consensus.cc:2454] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72caaa20959a43338ed1fb11bfac7520 in term 1.
I20260620 14:30:51.512315 8619 leader_election.cc:302] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b5d3e91ea449848d118842a30ccedb, 72caaa20959a43338ed1fb11bfac7520; no voters:
I20260620 14:30:51.512509 8886 raft_consensus.cc:2781] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:51.512719 8886 raft_consensus.cc:686] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 1 LEADER]: Becoming Leader. State: Replica: 72caaa20959a43338ed1fb11bfac7520, State: Running, Role: LEADER
I20260620 14:30:51.513139 8886 consensus_queue.cc:237] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } }
I20260620 14:30:51.516465 8405 catalog_manager.cc:5495] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72caaa20959a43338ed1fb11bfac7520 (127.4.58.66). New cstate: current_term: 1 leader_uuid: "72caaa20959a43338ed1fb11bfac7520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72caaa20959a43338ed1fb11bfac7520" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:51.533095 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.529591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.4.58.66:46643)
0620 14:30:51.529714 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.529718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.529732 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:30:51.529864 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.529867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.529931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.530028 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:51.530267 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.530763 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.531611 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:51.531843 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:51.531933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:51.532239 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:51.532687 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:51.532850 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.532882 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.532940 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:30:51.539191 8551 raft_consensus.cc:1261] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb [term 1 FOLLOWER]: Refusing update from remote peer 72caaa20959a43338ed1fb11bfac7520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:51.539192 8817 raft_consensus.cc:1261] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 1 FOLLOWER]: Refusing update from remote peer 72caaa20959a43338ed1fb11bfac7520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:51.539892 8890 consensus_queue.cc:1035] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b5d3e91ea449848d118842a30ccedb" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:30:51.540002 8864 tablet.cc:2369] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:51.540078 8886 consensus_queue.cc:1035] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39e78f386b749f891e9bab9fd8e9f9f" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:51.550489 8895 mvcc.cc:205] Tried to move back new op lower bound from 7298932127897161728 to 7298932127800524800. Current Snapshot: MvccSnapshot[applied={T|T < 7298932127897161728}]
I20260620 14:30:51.550828 8894 mvcc.cc:205] Tried to move back new op lower bound from 7298932127897161728 to 7298932127800524800. Current Snapshot: MvccSnapshot[applied={T|T < 7298932127897161728 or (T in {7298932127897161728})}]
I20260620 14:30:51.556600 8405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:51.556824 8405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:51.559372 8405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf5f4287e92a414eab058a4af5a22933: Sending DeleteTablet for 3 replicas of tablet 650c2f387a574aceae155a5eac425975
I20260620 14:30:51.560137 8664 tablet_service.cc:1514] Processing DeleteTablet for tablet 650c2f387a574aceae155a5eac425975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:51 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:36512
I20260620 14:30:51.560177 8531 tablet_service.cc:1514] Processing DeleteTablet for tablet 650c2f387a574aceae155a5eac425975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:51 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:56290
I20260620 14:30:51.560691 8904 tablet_replica.cc:330] T 650c2f387a574aceae155a5eac425975 P 56b5d3e91ea449848d118842a30ccedb: stopping tablet replica
I20260620 14:30:51.560694 8797 tablet_service.cc:1514] Processing DeleteTablet for tablet 650c2f387a574aceae155a5eac425975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:51 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:60894
I20260620 14:30:51.561033 8903 tablet_replica.cc:330] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: stopping tablet replica
I20260620 14:30:51.561198 8905 tablet_replica.cc:330] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: stopping tablet replica
I20260620 14:30:51.561511 8905 raft_consensus.cc:2229] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:51.561770 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8469
I20260620 14:30:51.561846 8905 raft_consensus.cc:2258] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:51.563041 8903 raft_consensus.cc:2229] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:51.563490 8903 raft_consensus.cc:2258] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:51.563603 8905 ts_tablet_manager.cc:1905] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:51.565146 8903 ts_tablet_manager.cc:1905] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:51.566504 8905 ts_tablet_manager.cc:1918] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:51.566608 8905 log.cc:1196] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/650c2f387a574aceae155a5eac425975
I20260620 14:30:51.566915 8905 ts_tablet_manager.cc:1939] T 650c2f387a574aceae155a5eac425975 P a39e78f386b749f891e9bab9fd8e9f9f: Deleting consensus metadata
I20260620 14:30:51.568120 8903 ts_tablet_manager.cc:1918] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:51.568284 8903 log.cc:1196] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/650c2f387a574aceae155a5eac425975
I20260620 14:30:51.568549 8394 catalog_manager.cc:4841] TS a39e78f386b749f891e9bab9fd8e9f9f (127.4.58.67:33563): tablet 650c2f387a574aceae155a5eac425975 (table test-table [id=92a1bdcd8fbe458ab220d6bd9ce04ce4]) successfully deleted
I20260620 14:30:51.568617 8903 ts_tablet_manager.cc:1939] T 650c2f387a574aceae155a5eac425975 P 72caaa20959a43338ed1fb11bfac7520: Deleting consensus metadata
I20260620 14:30:51.569908 8392 catalog_manager.cc:4841] TS 72caaa20959a43338ed1fb11bfac7520 (127.4.58.66:46643): tablet 650c2f387a574aceae155a5eac425975 (table test-table [id=92a1bdcd8fbe458ab220d6bd9ce04ce4]) successfully deleted
I20260620 14:30:51.570869 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8602
W20260620 14:30:51.570883 8393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:39819 (error 108)
I20260620 14:30:51.573024 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.572039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:39819 (local address 127.0.0.1:56302)
0620 14:30:51.572867 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:51.572953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748}
W20260620 14:30:51.573239 8393 catalog_manager.cc:4584] TS 56b5d3e91ea449848d118842a30ccedb (127.4.58.65:39819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 650c2f387a574aceae155a5eac425975: Network error: Client connection negotiation failed: client connection to 127.4.58.65:39819: connect: Connection refused (error 111)
I20260620 14:30:51.578097 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8735
I20260620 14:30:51.583217 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8376
2026-06-20T14:30:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2257 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:30:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:51Z Disabled control of system clock
I20260620 14:30:51.611763 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44923
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:30:51.722257 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:51.722544 8910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:51.722580 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:51.726527 8910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:51.726579 8910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:51.726594 8910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:51.726610 8910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:51.726624 8910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:51.731308 8910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:51.732357 8910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:51.733704 8910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:51.742072 8916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.742096 8915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.742077 8918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:51.742367 8910 server_base.cc:1034] running on GCE node
I20260620 14:30:51.743052 8910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:51.744114 8910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:51.745332 8910 hybrid_clock.cc:648] HybridClock initialized: now 1781965851745257 us; error 36 us; skew 500 ppm
I20260620 14:30:51.747278 8910 webserver.cc:460] Webserver started at http://127.4.58.126:43943/ using document root <none> and password file <none>
I20260620 14:30:51.747844 8910 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:51.747905 8910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:51.748122 8910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:51.750221 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "44563c2bfed54ae297d1e10de141977f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "9255d126114fc090a34fd6db38ea0927"
server_key_iv: "f7d42c7eb120a604316d0b357969a05a"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.750849 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "44563c2bfed54ae297d1e10de141977f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "9255d126114fc090a34fd6db38ea0927"
server_key_iv: "f7d42c7eb120a604316d0b357969a05a"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.754623 8910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:51.757177 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.758513 8910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:51.758647 8910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "44563c2bfed54ae297d1e10de141977f"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "9255d126114fc090a34fd6db38ea0927"
server_key_iv: "f7d42c7eb120a604316d0b357969a05a"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.758771 8910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:51.763377 8910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:51.763981 8910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:51.764132 8910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:51.770761 8910 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44923
I20260620 14:30:51.770784 8976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44923 every 8 connection(s)
I20260620 14:30:51.771759 8910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:51.775118 8977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:51.777347 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 8910
I20260620 14:30:51.777467 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:51.777750 4329 external_mini_cluster.cc:1428] Setting key b87ffb0c3b65eaba8965fcf112c0230d
I20260620 14:30:51.781778 8977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: Bootstrap starting.
I20260620 14:30:51.783879 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:51.779299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.4.58.126:44923)
0620 14:30:51.780044 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:51.780066 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:51.780111 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:30:51.780861 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:51.780883 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:51.781390 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:51.781751 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:51.782581 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:51.782586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:51.782598 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:51.782610 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:51.782674 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:51.782693 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:51.782849 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:51.782926 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:30:51.783175 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
I20260620 14:30:51.784145 8977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:51.784929 8977 log.cc:824] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:51.786820 8977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: No bootstrap required, opened a new log
I20260620 14:30:51.790460 8977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } }
I20260620 14:30:51.790680 8977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:51.790742 8977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44563c2bfed54ae297d1e10de141977f, State: Initialized, Role: FOLLOWER
I20260620 14:30:51.791162 8977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } }
I20260620 14:30:51.791301 8977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:51.791363 8977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:51.791461 8977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:51.792309 8977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } }
I20260620 14:30:51.792621 8977 leader_election.cc:302] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44563c2bfed54ae297d1e10de141977f; no voters:
I20260620 14:30:51.792937 8977 leader_election.cc:288] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:51.793061 8982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:51.793220 8982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [term 1 LEADER]: Becoming Leader. State: Replica: 44563c2bfed54ae297d1e10de141977f, State: Running, Role: LEADER
I20260620 14:30:51.793506 8982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } }
I20260620 14:30:51.794236 8977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:51.795120 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44563c2bfed54ae297d1e10de141977f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } } }
I20260620 14:30:51.795238 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:51.795569 8984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44563c2bfed54ae297d1e10de141977f. Latest consensus state: current_term: 1 leader_uuid: "44563c2bfed54ae297d1e10de141977f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44563c2bfed54ae297d1e10de141977f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44923 } } }
I20260620 14:30:51.795656 8984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:51.796382 8991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:51.798982 8991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:51.804416 8991 catalog_manager.cc:1347] Generated new cluster ID: c1f30118b0c44b5884f0e45bb0bc9314
I20260620 14:30:51.804486 8991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:51.824803 8991 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:51.825194 8991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:51.832500 8991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: Generated new TSK 0
I20260620 14:30:51.833420 8991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:51.843252 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--builtin_ntp_servers=127.4.58.84:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:30:51.953796 9001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:51.954077 9001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:51.954133 9001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:51.958184 9001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:51.958243 9001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:51.958319 9001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:51.963201 9001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:51.964260 9001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:51.965682 9001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:51.979648 9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.979691 9009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:51.979705 9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:51.979754 9001 server_base.cc:1034] running on GCE node
I20260620 14:30:51.980310 9001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:51.981033 9001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:51.982195 9001 hybrid_clock.cc:648] HybridClock initialized: now 1781965851982148 us; error 62 us; skew 500 ppm
I20260620 14:30:51.984197 9001 webserver.cc:460] Webserver started at http://127.4.58.65:40421/ using document root <none> and password file <none>
I20260620 14:30:51.984906 9001 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:51.984985 9001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:51.985203 9001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:51.987414 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "13df10e5556992db1b571f8d81769d22"
server_key_iv: "6c0ec9ae197d2acd6e253f4f5e4e202f"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.988005 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "13df10e5556992db1b571f8d81769d22"
server_key_iv: "6c0ec9ae197d2acd6e253f4f5e4e202f"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.991894 9001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:30:51.994480 9015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:51.995900 9001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:30:51.996008 9001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7"
format_stamp: "Formatted at 2026-06-20 14:30:51 on dist-test-slave-gqzk"
server_key: "13df10e5556992db1b571f8d81769d22"
server_key_iv: "6c0ec9ae197d2acd6e253f4f5e4e202f"
server_key_version: "encryptionkey@0"
I20260620 14:30:51.996105 9001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:52.000586 9001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:52.001233 9001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:52.001396 9001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:52.001945 9001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:52.002847 9001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:52.002918 9001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.002970 9001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:52.003000 9001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.011606 9001 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:44549
I20260620 14:30:52.011633 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:44549 every 8 connection(s)
I20260620 14:30:52.012535 9001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:52.016356 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9001
I20260620 14:30:52.016477 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:52.016808 4329 external_mini_cluster.cc:1428] Setting key 39f53acf7f43b8f1317d35a7ab5cb708
I20260620 14:30:52.018795 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--builtin_ntp_servers=127.4.58.84:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:30:52.019042 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.014409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:39995 (local address 127.4.58.126:44923)
0620 14:30:52.014536 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.014539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.015846 (+ 1307us) server_negotiation.cc:407] Connection header received
0620 14:30:52.017350 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.017354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.017408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.017480 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.018605 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.018608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.018610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.018620 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.018641 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.018648 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.018720 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.018849 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.018913 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:30:52.019930 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.014831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44923 (local address 127.4.58.65:39995)
0620 14:30:52.015654 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.015710 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.017125 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.017662 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.017670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.017764 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.018341 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.018343 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.018393 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.018465 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.018715 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.018724 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.018730 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.018892 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.019147 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":162,"threads_started":1}
I20260620 14:30:52.021255 9129 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44923
I20260620 14:30:52.021484 9129 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:52.021903 9129 heartbeater.cc:511] Master 127.4.58.126:44923 requested a full tablet report, sending...
I20260620 14:30:52.023660 8941 ts_manager.cc:194] Registered new tserver with Master: 9b9f7893ed1d4e50bb9d5d2e3872aeb7 (127.4.58.65:44549)
W20260620 14:30:52.131309 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:52.131585 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:52.131623 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:52.136073 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:52.136143 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:52.136205 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:52.141029 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:52.142056 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:52.143378 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:52.152509 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:52.152520 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:52.152542 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:52.152968 9132 server_base.cc:1034] running on GCE node
I20260620 14:30:52.153365 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:52.153965 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:52.155126 9132 hybrid_clock.cc:648] HybridClock initialized: now 1781965852155107 us; error 30 us; skew 500 ppm
I20260620 14:30:52.156936 9132 webserver.cc:460] Webserver started at http://127.4.58.66:42541/ using document root <none> and password file <none>
I20260620 14:30:52.157492 9132 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:52.157557 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:52.157743 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:52.160053 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "5577374e024b4b0a9b540e66278c40b1"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "414d7f65e6af0c7045602855ccda03db"
server_key_iv: "f3475c981e809637f9e539bd1de4f478"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.160579 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "5577374e024b4b0a9b540e66278c40b1"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "414d7f65e6af0c7045602855ccda03db"
server_key_iv: "f3475c981e809637f9e539bd1de4f478"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.164283 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:30:52.166630 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.167843 9132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:52.167945 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "5577374e024b4b0a9b540e66278c40b1"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "414d7f65e6af0c7045602855ccda03db"
server_key_iv: "f3475c981e809637f9e539bd1de4f478"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.168040 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:52.172425 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:52.173053 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:52.173202 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:52.173710 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:52.174583 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:52.174629 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.174677 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:52.174705 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.183005 9132 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:36337
I20260620 14:30:52.183038 9259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:36337 every 8 connection(s)
I20260620 14:30:52.183897 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:52.185364 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9132
I20260620 14:30:52.185463 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:52.185736 4329 external_mini_cluster.cc:1428] Setting key 6b67554fcc85265a6f4a027fe6f029f1
I20260620 14:30:52.187600 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--builtin_ntp_servers=127.4.58.84:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:30:52.190471 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.185957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:52621 (local address 127.4.58.126:44923)
0620 14:30:52.186110 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.186116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.187416 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 14:30:52.188733 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.188736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.188805 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.188882 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.190038 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.190042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.190045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.190058 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.190082 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.190091 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.190180 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.190283 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.190327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:30:52.191418 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.186488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44923 (local address 127.4.58.66:52621)
0620 14:30:52.187251 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.187304 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.188519 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.189069 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.189076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.189169 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.189740 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.189742 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.189794 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.189875 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.190172 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.190178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.190181 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.190382 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.190670 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260620 14:30:52.192641 9260 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44923
I20260620 14:30:52.192886 9260 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:52.193284 9260 heartbeater.cc:511] Master 127.4.58.126:44923 requested a full tablet report, sending...
I20260620 14:30:52.194416 8941 ts_manager.cc:194] Registered new tserver with Master: 5577374e024b4b0a9b540e66278c40b1 (127.4.58.66:36337)
W20260620 14:30:52.298482 9263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:52.298768 9263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:52.298828 9263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:52.303066 9263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:52.303134 9263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:52.303216 9263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:52.308120 9263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:52.309254 9263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:52.310609 9263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:52.320116 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:52.320120 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:52.320170 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:52.320508 9263 server_base.cc:1034] running on GCE node
I20260620 14:30:52.320945 9263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:52.321605 9263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:52.322768 9263 hybrid_clock.cc:648] HybridClock initialized: now 1781965852322744 us; error 40 us; skew 500 ppm
I20260620 14:30:52.324852 9263 webserver.cc:460] Webserver started at http://127.4.58.67:45025/ using document root <none> and password file <none>
I20260620 14:30:52.325490 9263 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:52.325565 9263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:52.325781 9263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:52.327870 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "9d0d8143b0e243d7ae26ab265fc3acb3"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "a6f2a4e67a2884d8e8d627742c1240ed"
server_key_iv: "f98eeaedbc731da34882c2953a00374d"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.328433 9263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "9d0d8143b0e243d7ae26ab265fc3acb3"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "a6f2a4e67a2884d8e8d627742c1240ed"
server_key_iv: "f98eeaedbc731da34882c2953a00374d"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.332281 9263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:30:52.335039 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.336369 9263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:30:52.336517 9263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "9d0d8143b0e243d7ae26ab265fc3acb3"
format_stamp: "Formatted at 2026-06-20 14:30:52 on dist-test-slave-gqzk"
server_key: "a6f2a4e67a2884d8e8d627742c1240ed"
server_key_iv: "f98eeaedbc731da34882c2953a00374d"
server_key_version: "encryptionkey@0"
I20260620 14:30:52.336628 9263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:52.341730 9263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:52.342321 9263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:52.342480 9263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:52.343012 9263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:52.343891 9263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:52.343945 9263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.343995 9263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:52.344022 9263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:52.352478 9263 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:45049
I20260620 14:30:52.352541 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:45049 every 8 connection(s)
I20260620 14:30:52.353467 9263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:52.354018 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9263
I20260620 14:30:52.354133 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:52.354422 4329 external_mini_cluster.cc:1428] Setting key 8cd88ecc5002aef2c2fc0d5e06386ac7
I20260620 14:30:52.359678 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.355366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:38379 (local address 127.4.58.126:44923)
0620 14:30:52.355496 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.355500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.356901 (+ 1401us) server_negotiation.cc:407] Connection header received
0620 14:30:52.358179 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.358182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.358216 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.358274 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.359309 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.359312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.359314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.359326 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.359346 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.359354 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.359436 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.359531 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.359569 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:52.360554 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.355949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44923 (local address 127.4.58.67:38379)
0620 14:30:52.356727 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.356805 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.357979 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.358441 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.358447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.358551 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.359064 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.359065 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.359107 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.359179 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.359418 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.359423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.359426 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.359564 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.359765 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":144,"threads_started":1}
I20260620 14:30:52.361624 9391 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44923
I20260620 14:30:52.361868 9391 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:52.362290 9391 heartbeater.cc:511] Master 127.4.58.126:44923 requested a full tablet report, sending...
I20260620 14:30:52.363308 8941 ts_manager.cc:194] Registered new tserver with Master: 9d0d8143b0e243d7ae26ab265fc3acb3 (127.4.58.67:45049)
I20260620 14:30:52.368513 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:30:52.371909 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.370362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.4.58.126:44923)
0620 14:30:52.370546 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.370551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.370678 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:30:52.370809 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.370811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.370844 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.370906 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.371540 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.371544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.371546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.371558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.371578 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.371585 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.371655 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.371739 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.371781 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:30:52.374475 8941 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:52.377504 8941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:52.389871 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.385544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36337 (local address 127.0.0.1:57120)
0620 14:30:52.386157 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.386206 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.387018 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.388149 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.388152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.388203 (+ 51us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.388924 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.388926 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.388937 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.389012 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.389619 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.389622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.389626 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.389686 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.389722 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":91,"threads_started":1}
I20260620 14:30:52.390244 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.386343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.4.58.65:44549)
0620 14:30:52.386947 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.386957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.386995 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:52.387051 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.387057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.387262 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.387397 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.389234 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.389242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.389255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.389268 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.389319 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.389380 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.389820 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.389884 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.390070 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":86,"threads_started":1}
I20260620 14:30:52.390758 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.385684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44549 (local address 127.0.0.1:36934)
0620 14:30:52.386774 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.386788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.386893 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.388116 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.388124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.388204 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.388886 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.388888 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.388910 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.389037 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.389489 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.389500 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.389507 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.389619 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.389690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":91,"threads_started":1}
I20260620 14:30:52.392182 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.385669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.4.58.66:36337)
0620 14:30:52.387772 (+ 2103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.387782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.387804 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:30:52.387852 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.387857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.388017 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.388806 (+ 789us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.389416 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.389421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.389432 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.389443 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.389484 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.389512 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.391879 (+ 2367us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.391931 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.392018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":97,"threads_started":1}
I20260620 14:30:52.393438 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.385897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:45049 (local address 127.0.0.1:44066)
0620 14:30:52.387077 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.387090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.387158 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.388214 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.388217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.388236 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.390927 (+ 2691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.390929 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.390941 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.390969 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.393194 (+ 2225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.393198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.393203 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.393275 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.393315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":90,"threads_started":1}
I20260620 14:30:52.393937 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.386323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.4.58.67:45049)
0620 14:30:52.386760 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.386772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.387249 (+ 477us) server_negotiation.cc:407] Connection header received
0620 14:30:52.387304 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.387309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.387478 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.387598 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.392897 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.392904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.392918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.392934 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.392982 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.393030 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.393145 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.393681 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.393761 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260620 14:30:52.394125 9063 tablet_service.cc:1467] Processing CreateTablet for tablet bcd39a5509b74fc9b75e40d82680e547 (DEFAULT_TABLE table=test-table [id=0efe37898fbd46f9802db2ec7a89765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:52.395120 9194 tablet_service.cc:1467] Processing CreateTablet for tablet bcd39a5509b74fc9b75e40d82680e547 (DEFAULT_TABLE table=test-table [id=0efe37898fbd46f9802db2ec7a89765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:52.396292 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd39a5509b74fc9b75e40d82680e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:52.396293 9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd39a5509b74fc9b75e40d82680e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:52.397343 9325 tablet_service.cc:1467] Processing CreateTablet for tablet bcd39a5509b74fc9b75e40d82680e547 (DEFAULT_TABLE table=test-table [id=0efe37898fbd46f9802db2ec7a89765a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:52.398213 9325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd39a5509b74fc9b75e40d82680e547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:52.405143 9408 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Bootstrap starting.
I20260620 14:30:52.405577 9410 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Bootstrap starting.
I20260620 14:30:52.407352 9410 tablet_bootstrap.cc:654] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:52.407437 9408 tablet_bootstrap.cc:654] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:52.407702 9409 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Bootstrap starting.
I20260620 14:30:52.408160 9410 log.cc:824] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:52.408224 9408 log.cc:824] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:52.409803 9409 tablet_bootstrap.cc:654] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:52.410301 9410 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: No bootstrap required, opened a new log
I20260620 14:30:52.410496 9410 ts_tablet_manager.cc:1397] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:30:52.410580 9409 log.cc:824] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:52.411188 9408 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: No bootstrap required, opened a new log
I20260620 14:30:52.411367 9408 ts_tablet_manager.cc:1397] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 14:30:52.412369 9409 tablet_bootstrap.cc:492] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: No bootstrap required, opened a new log
I20260620 14:30:52.412561 9409 ts_tablet_manager.cc:1397] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:30:52.414721 9408 raft_consensus.cc:348] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.414942 9408 raft_consensus.cc:374] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:52.415004 9408 raft_consensus.cc:729] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5577374e024b4b0a9b540e66278c40b1, State: Initialized, Role: FOLLOWER
I20260620 14:30:52.414954 9410 raft_consensus.cc:348] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.415225 9410 raft_consensus.cc:374] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:52.415270 9410 raft_consensus.cc:729] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0d8143b0e243d7ae26ab265fc3acb3, State: Initialized, Role: FOLLOWER
I20260620 14:30:52.415442 9408 consensus_queue.cc:260] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.415688 9410 consensus_queue.cc:260] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.416157 9260 heartbeater.cc:503] Master 127.4.58.126:44923 was elected leader, sending a full tablet report...
I20260620 14:30:52.416474 9408 ts_tablet_manager.cc:1428] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 14:30:52.416621 9410 ts_tablet_manager.cc:1428] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:30:52.416965 9391 heartbeater.cc:503] Master 127.4.58.126:44923 was elected leader, sending a full tablet report...
I20260620 14:30:52.416826 9409 raft_consensus.cc:348] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.417078 9409 raft_consensus.cc:374] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:52.417116 9409 raft_consensus.cc:729] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9f7893ed1d4e50bb9d5d2e3872aeb7, State: Initialized, Role: FOLLOWER
I20260620 14:30:52.417521 9409 consensus_queue.cc:260] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.418387 9409 ts_tablet_manager.cc:1428] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:30:52.418718 9129 heartbeater.cc:503] Master 127.4.58.126:44923 was elected leader, sending a full tablet report...
W20260620 14:30:52.434799 9261 tablet.cc:2369] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:52.513705 9130 tablet.cc:2369] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:52.604442 9392 tablet.cc:2369] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:52.792054 9415 raft_consensus.cc:482] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:52.792253 9415 raft_consensus.cc:504] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.793471 9415 leader_election.cc:288] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5577374e024b4b0a9b540e66278c40b1 (127.4.58.66:36337), 9b9f7893ed1d4e50bb9d5d2e3872aeb7 (127.4.58.65:44549)
I20260620 14:30:52.795447 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.793741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44549 (local address 127.4.58.67:43515)
0620 14:30:52.793844 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.793859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.793948 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.794272 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.794275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.794300 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.794790 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.794792 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.794803 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.794833 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.795209 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.795212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.795215 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.795287 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.795322 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 14:30:52.795552 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.793834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:43229 (local address 127.4.58.66:36337)
0620 14:30:52.793990 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.793995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.794359 (+ 364us) server_negotiation.cc:407] Connection header received
0620 14:30:52.794440 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.794442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.794476 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.794558 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.795021 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.795024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.795027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.795042 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.795067 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.795079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.795188 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.795399 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.795440 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:30:52.795537 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.793834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:43515 (local address 127.4.58.65:44549)
0620 14:30:52.793974 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.793979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.794028 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:30:52.794097 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.794100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.794141 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.794222 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.795020 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.795024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.795027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.795042 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.795067 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.795079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.795189 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.795348 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.795393 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:30:52.795791 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.793741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36337 (local address 127.4.58.67:43229)
0620 14:30:52.794271 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.794283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:52.794366 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:52.794577 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:52.794579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:52.794591 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:52.794867 (+ 276us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:30:52.794869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:52.794876 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:30:52.794896 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:52.795238 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:52.795242 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:52.795246 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:30:52.795339 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:30:52.795379 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":164,"thread_start_us":146,"threads_started":1}
I20260620 14:30:52.796262 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd39a5509b74fc9b75e40d82680e547" candidate_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" is_pre_election: true
I20260620 14:30:52.796386 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd39a5509b74fc9b75e40d82680e547" candidate_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5577374e024b4b0a9b540e66278c40b1" is_pre_election: true
I20260620 14:30:52.796628 9083 raft_consensus.cc:2454] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0d8143b0e243d7ae26ab265fc3acb3 in term 0.
I20260620 14:30:52.796655 9214 raft_consensus.cc:2454] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0d8143b0e243d7ae26ab265fc3acb3 in term 0.
I20260620 14:30:52.797151 9278 leader_election.cc:302] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5577374e024b4b0a9b540e66278c40b1, 9b9f7893ed1d4e50bb9d5d2e3872aeb7, 9d0d8143b0e243d7ae26ab265fc3acb3; no voters:
I20260620 14:30:52.797427 9415 raft_consensus.cc:2781] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:52.797515 9415 raft_consensus.cc:482] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:52.797566 9415 raft_consensus.cc:3037] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:52.798626 9415 raft_consensus.cc:504] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.799000 9415 leader_election.cc:288] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [CANDIDATE]: Term 1 election: Requested vote from peers 5577374e024b4b0a9b540e66278c40b1 (127.4.58.66:36337), 9b9f7893ed1d4e50bb9d5d2e3872aeb7 (127.4.58.65:44549)
I20260620 14:30:52.799418 9214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd39a5509b74fc9b75e40d82680e547" candidate_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5577374e024b4b0a9b540e66278c40b1"
I20260620 14:30:52.799420 9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd39a5509b74fc9b75e40d82680e547" candidate_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7"
I20260620 14:30:52.799551 9083 raft_consensus.cc:3037] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:52.799551 9214 raft_consensus.cc:3037] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:52.800840 9083 raft_consensus.cc:2454] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0d8143b0e243d7ae26ab265fc3acb3 in term 1.
I20260620 14:30:52.800840 9214 raft_consensus.cc:2454] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0d8143b0e243d7ae26ab265fc3acb3 in term 1.
I20260620 14:30:52.801156 9279 leader_election.cc:302] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9f7893ed1d4e50bb9d5d2e3872aeb7, 9d0d8143b0e243d7ae26ab265fc3acb3; no voters:
I20260620 14:30:52.801329 9415 raft_consensus.cc:2781] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:52.801517 9415 raft_consensus.cc:686] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 1 LEADER]: Becoming Leader. State: Replica: 9d0d8143b0e243d7ae26ab265fc3acb3, State: Running, Role: LEADER
I20260620 14:30:52.801859 9415 consensus_queue.cc:237] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } }
I20260620 14:30:52.804936 8940 catalog_manager.cc:5495] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d0d8143b0e243d7ae26ab265fc3acb3 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d0d8143b0e243d7ae26ab265fc3acb3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 45049 } health_report { overall_health: HEALTHY } } }
I20260620 14:30:52.850210 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.844213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.4.58.67:45049)
0620 14:30:52.844353 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:52.844357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:52.844391 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:30:52.844565 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:52.844568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:52.844631 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:52.844704 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:52.849857 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:52.849861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:52.849863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:52.849878 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:52.849899 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:52.849908 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:52.849999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:52.850059 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:30:52.850097 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:30:52.856165 9083 raft_consensus.cc:1261] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 1 FOLLOWER]: Refusing update from remote peer 9d0d8143b0e243d7ae26ab265fc3acb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:52.856168 9214 raft_consensus.cc:1261] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 1 FOLLOWER]: Refusing update from remote peer 9d0d8143b0e243d7ae26ab265fc3acb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:52.856935 9418 consensus_queue.cc:1035] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5577374e024b4b0a9b540e66278c40b1" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:52.857152 9415 consensus_queue.cc:1035] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9f7893ed1d4e50bb9d5d2e3872aeb7" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:52.865695 9422 mvcc.cc:205] Tried to move back new op lower bound from 7298932133291769856 to 7298932133078806528. Current Snapshot: MvccSnapshot[applied={T|T < 7298932133291769856}]
I20260620 14:30:52.874284 8941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:52.874495 8941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:52.877373 8941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44563c2bfed54ae297d1e10de141977f: Sending DeleteTablet for 3 replicas of tablet bcd39a5509b74fc9b75e40d82680e547
I20260620 14:30:52.878239 9063 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd39a5509b74fc9b75e40d82680e547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:52 UTC) from {username='slave'} at 127.0.0.1:36934
I20260620 14:30:52.878294 9325 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd39a5509b74fc9b75e40d82680e547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:52 UTC) from {username='slave'} at 127.0.0.1:44066
I20260620 14:30:52.878333 9194 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd39a5509b74fc9b75e40d82680e547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:52 UTC) from {username='slave'} at 127.0.0.1:57120
I20260620 14:30:52.878772 9432 tablet_replica.cc:330] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: stopping tablet replica
I20260620 14:30:52.878983 9431 tablet_replica.cc:330] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: stopping tablet replica
I20260620 14:30:52.879130 9432 raft_consensus.cc:2229] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:30:52.879220 9433 tablet_replica.cc:330] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7: stopping tablet replica
I20260620 14:30:52.879520 9433 raft_consensus.cc:2229] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:52.879559 9432 raft_consensus.cc:2258] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:52.879686 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9001
I20260620 14:30:52.879844 9433 raft_consensus.cc:2258] T bcd39a5509b74fc9b75e40d82680e547 P 9b9f7893ed1d4e50bb9d5d2e3872aeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:52.881170 9432 ts_tablet_manager.cc:1905] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:52.882244 9431 raft_consensus.cc:2229] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:52.882651 9431 raft_consensus.cc:2258] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:52.884003 9432 ts_tablet_manager.cc:1918] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:52.884094 9432 log.cc:1196] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/bcd39a5509b74fc9b75e40d82680e547
I20260620 14:30:52.884375 9432 ts_tablet_manager.cc:1939] T bcd39a5509b74fc9b75e40d82680e547 P 9d0d8143b0e243d7ae26ab265fc3acb3: Deleting consensus metadata
I20260620 14:30:52.885517 9431 ts_tablet_manager.cc:1905] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:52.885596 8928 catalog_manager.cc:4841] TS 9d0d8143b0e243d7ae26ab265fc3acb3 (127.4.58.67:45049): tablet bcd39a5509b74fc9b75e40d82680e547 (table test-table [id=0efe37898fbd46f9802db2ec7a89765a]) successfully deleted
W20260620 14:30:52.887109 8926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:44549 (error 108)
I20260620 14:30:52.888126 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.887827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44549 (local address 127.0.0.1:36938)
0620 14:30:52.887965 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.888036 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:30:52.888221 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9132
I20260620 14:30:52.888366 9431 ts_tablet_manager.cc:1918] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:30:52.888454 9431 log.cc:1196] T bcd39a5509b74fc9b75e40d82680e547 P 5577374e024b4b0a9b540e66278c40b1: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/bcd39a5509b74fc9b75e40d82680e547
W20260620 14:30:52.889518 8926 catalog_manager.cc:4584] TS 9b9f7893ed1d4e50bb9d5d2e3872aeb7 (127.4.58.65:44549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcd39a5509b74fc9b75e40d82680e547: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44549: connect: Connection refused (error 111)
I20260620 14:30:52.893956 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9263
I20260620 14:30:52.895531 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:52.894195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36337 (local address 127.0.0.1:57124)
0620 14:30:52.895372 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:52.895446 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:36337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1086}
I20260620 14:30:52.899109 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 8910
2026-06-20T14:30:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1314 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:30:52.919116 4329 mini_oidc.cc:152] Starting JWKS server
I20260620 14:30:52.920682 4329 webserver.cc:460] Webserver started at https://127.0.0.1:46733/ using document root <none> and password file <none>
I20260620 14:30:52.920879 4329 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:30:52.921121 4329 webserver.cc:460] Webserver started at http://127.0.0.1:43601/ using document root <none> and password file <none>
2026-06-20T14:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:52Z Disabled control of system clock
I20260620 14:30:52.929579 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44391
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44391
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46733/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/testchainca.cert with env {}
W20260620 14:30:53.038321 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:53.038612 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:53.038652 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:53.042737 9440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:53.042805 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:53.042819 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:53.042835 9440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:53.042850 9440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:30:53.042865 9440 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:30:53.042876 9440 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46733/jwks.json
I20260620 14:30:53.048377 9440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46733/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:53.049467 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:53.050436 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:53.058672 9446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:53.058729 9445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:53.058719 9448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:53.058810 9440 server_base.cc:1034] running on GCE node
I20260620 14:30:53.059513 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:53.060667 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:53.061841 9440 hybrid_clock.cc:648] HybridClock initialized: now 1781965853061807 us; error 48 us; skew 500 ppm
I20260620 14:30:53.063750 9440 webserver.cc:460] Webserver started at http://127.4.58.126:45843/ using document root <none> and password file <none>
I20260620 14:30:53.064337 9440 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:53.064406 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:53.064590 9440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:53.066673 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "f43a729f47a74159b0281c462da8fd3c"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "a72cd3789bba9bbef531665cb4c60ac0"
server_key_iv: "1cfa5d4935227ca749b3e4e91bd2fbdc"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.067226 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "f43a729f47a74159b0281c462da8fd3c"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "a72cd3789bba9bbef531665cb4c60ac0"
server_key_iv: "1cfa5d4935227ca749b3e4e91bd2fbdc"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.070914 9440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:30:53.073510 9454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:53.074832 9440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:53.074973 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "f43a729f47a74159b0281c462da8fd3c"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "a72cd3789bba9bbef531665cb4c60ac0"
server_key_iv: "1cfa5d4935227ca749b3e4e91bd2fbdc"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.075074 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:53.099740 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:53.100493 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:53.100680 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:53.107831 9440 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44391
I20260620 14:30:53.107895 9506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44391 every 8 connection(s)
I20260620 14:30:53.108824 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:53.112058 9507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:53.115367 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9440
I20260620 14:30:53.115510 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:53.115794 4329 external_mini_cluster.cc:1428] Setting key 8d06f952b190b194df1b4c769eec20ea
I20260620 14:30:53.118450 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c: Bootstrap starting.
I20260620 14:30:53.120632 9507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:53.121469 9507 log.cc:824] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:53.123462 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c: No bootstrap required, opened a new log
I20260620 14:30:53.125317 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.117282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.4.58.126:44391)
0620 14:30:53.118028 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.118045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.118081 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:30:53.118765 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.118785 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.119210 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.119545 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:53.120002 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.121064 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.121978 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.122466 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.122614 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.123504 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:53.123508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:53.123522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:53.123944 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:53.124003 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:53.124023 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:53.124105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:53.124248 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:30:53.124514 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":128,"threads_started":1}
I20260620 14:30:53.127105 9507 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } }
I20260620 14:30:53.127348 9507 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:53.127399 9507 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43a729f47a74159b0281c462da8fd3c, State: Initialized, Role: FOLLOWER
I20260620 14:30:53.127816 9507 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } }
I20260620 14:30:53.127964 9507 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:53.128034 9507 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:53.128139 9507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:53.128190 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:53.129158 9507 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } }
I20260620 14:30:53.129500 9507 leader_election.cc:302] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43a729f47a74159b0281c462da8fd3c; no voters:
I20260620 14:30:53.129806 9507 leader_election.cc:288] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:53.129901 9512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:53.130064 9512 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [term 1 LEADER]: Becoming Leader. State: Replica: f43a729f47a74159b0281c462da8fd3c, State: Running, Role: LEADER
I20260620 14:30:53.130386 9512 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } }
I20260620 14:30:53.131143 9507 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:53.132015 9513 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43a729f47a74159b0281c462da8fd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } } }
I20260620 14:30:53.132332 9514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43a729f47a74159b0281c462da8fd3c. Latest consensus state: current_term: 1 leader_uuid: "f43a729f47a74159b0281c462da8fd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43a729f47a74159b0281c462da8fd3c" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44391 } } }
I20260620 14:30:53.132477 9514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.132437 9513 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.133749 9522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:53.136319 9522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:53.141894 9522 catalog_manager.cc:1347] Generated new cluster ID: bb838565fe8d486bad122ca55e89e3e2
I20260620 14:30:53.141964 9522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:53.168355 9522 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:53.169409 9522 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:53.177131 9522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43a729f47a74159b0281c462da8fd3c: Generated new TSK 0
I20260620 14:30:53.177882 9522 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:30:53.207173 9537 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:30:53.207284 9537 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:30:53.207918 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.190281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.4.58.126:44391)
0620 14:30:53.190420 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.190425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.190587 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:30:53.190785 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.190789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.190850 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.190968 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:30:53.191258 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.192097 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.193080 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.193443 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.193547 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.193919 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:30:53.207459 (+ 13540us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:30:53.207757 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 14:30:53.208014 9510 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:30:53.209203 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:53.190175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44391 (local address 127.0.0.1:51972)
0620 14:30:53.190500 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:53.190513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:53.190627 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:53.191024 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:53.191027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:53.191035 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:30:53.191145 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.191150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.192234 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.192238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.192967 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.192977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.193582 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.193585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.193668 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.193717 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:30:53.207702 (+ 13985us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:30:53.207951 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.58.126:44391: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.58.126:44391: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46733/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:30:53.209919 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9440
2026-06-20T14:30:53Z chronyd exiting
I20260620 14:30:53.246861 4329 test_util.cc:175] -----------------------------------------------
I20260620 14:30:53.246949 4329 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965824702348-4329-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (329 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:30:53.247931 4329 mini_oidc.cc:152] Starting JWKS server
I20260620 14:30:53.248776 4329 webserver.cc:460] Webserver started at https://127.0.0.1:42527/ using document root <none> and password file <none>
I20260620 14:30:53.248895 4329 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:30:53.249146 4329 webserver.cc:460] Webserver started at http://127.0.0.1:44261/ using document root <none> and password file <none>
2026-06-20T14:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:53Z Disabled control of system clock
I20260620 14:30:53.257241 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45953
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:45025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45953
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/testchainca.cert with env {}
W20260620 14:30:53.368530 9543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:53.368842 9543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:53.368903 9543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:53.372913 9543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:53.372982 9543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:53.373004 9543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:53.373026 9543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:53.373044 9543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:30:53.373066 9543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:30:53.373093 9543 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42527/jwks.json
I20260620 14:30:53.378600 9543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:53.379693 9543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:53.380731 9543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:53.389115 9548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:53.389155 9549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:53.389155 9543 server_base.cc:1034] running on GCE node
W20260620 14:30:53.389423 9551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:53.390034 9543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:53.391196 9543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:53.392377 9543 hybrid_clock.cc:648] HybridClock initialized: now 1781965853392361 us; error 29 us; skew 500 ppm
I20260620 14:30:53.394451 9543 webserver.cc:460] Webserver started at http://127.4.58.126:44209/ using document root <none> and password file <none>
I20260620 14:30:53.395016 9543 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:53.395077 9543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:53.395237 9543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:53.397397 9543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "6eba3402c7ecb3e376b16fe322592c59"
server_key_iv: "02714cd3f58e42945a8f06d36f18c011"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.397919 9543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "6eba3402c7ecb3e376b16fe322592c59"
server_key_iv: "02714cd3f58e42945a8f06d36f18c011"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.401748 9543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:30:53.404204 9557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:53.405530 9543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:53.405651 9543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "6eba3402c7ecb3e376b16fe322592c59"
server_key_iv: "02714cd3f58e42945a8f06d36f18c011"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.405757 9543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:53.423184 9543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:53.423946 9543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:53.424130 9543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:53.431185 9543 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:45953
I20260620 14:30:53.431236 9609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:45953 every 8 connection(s)
I20260620 14:30:53.432178 9543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:53.433017 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9543
I20260620 14:30:53.433132 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:53.433441 4329 external_mini_cluster.cc:1428] Setting key 44901e28edc699c95c9b45c908730673
I20260620 14:30:53.435887 9610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:53.441540 9610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb: Bootstrap starting.
I20260620 14:30:53.442600 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.434811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.4.58.126:45953)
0620 14:30:53.435516 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.435531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.435568 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:53.436290 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.436316 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.436725 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.437112 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:53.437549 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.438541 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.439449 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.439934 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.440079 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.440918 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:53.440923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:53.440936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:53.441330 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:53.441388 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:53.441407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:53.441491 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:53.441631 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:30:53.441896 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":128,"threads_started":1}
I20260620 14:30:53.443962 9610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:53.444828 9610 log.cc:824] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:53.445561 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:53.446964 9610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb: No bootstrap required, opened a new log
I20260620 14:30:53.450558 9610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } }
I20260620 14:30:53.450800 9610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:53.450848 9610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cf6da76f3e4e8d8d97faa6b5235ddb, State: Initialized, Role: FOLLOWER
I20260620 14:30:53.451294 9610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } }
I20260620 14:30:53.451419 9610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:53.451480 9610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:53.451573 9610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:53.452626 9610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } }
I20260620 14:30:53.452986 9610 leader_election.cc:302] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53cf6da76f3e4e8d8d97faa6b5235ddb; no voters:
I20260620 14:30:53.453303 9610 leader_election.cc:288] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:53.453462 9615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:53.453683 9615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [term 1 LEADER]: Becoming Leader. State: Replica: 53cf6da76f3e4e8d8d97faa6b5235ddb, State: Running, Role: LEADER
I20260620 14:30:53.453976 9615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } }
I20260620 14:30:53.454519 9610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:53.455899 9616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } } }
I20260620 14:30:53.455897 9617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53cf6da76f3e4e8d8d97faa6b5235ddb. Latest consensus state: current_term: 1 leader_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53cf6da76f3e4e8d8d97faa6b5235ddb" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45953 } } }
I20260620 14:30:53.456051 9616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.456051 9617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.456983 9624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:53.459733 9624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:53.465312 9624 catalog_manager.cc:1347] Generated new cluster ID: d781a27bf68446189cd3d6d8606a3e85
I20260620 14:30:53.465440 9624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:53.477674 9624 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:53.478703 9624 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:53.485649 9624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53cf6da76f3e4e8d8d97faa6b5235ddb: Generated new TSK 0
I20260620 14:30:53.486389 9624 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:30:53.510578 9640 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:30:53.510679 9640 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:30:53.511407 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.495645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.4.58.126:45953)
0620 14:30:53.495773 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.495778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.495970 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:30:53.496148 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.496150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.496200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.496298 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:30:53.496565 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.497154 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.498019 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.498226 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.498310 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.498616 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:30:53.510956 (+ 12340us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:30:53.511239 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:30:53.511483 9613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:30:53.511514 9639 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:53.495568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45953 (local address 127.0.0.1:53480)
0620 14:30:53.495885 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:53.495897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:53.496009 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:53.496320 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:53.496324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:53.496331 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:30:53.496456 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.496463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.497271 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.497275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.497906 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.497915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.498329 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.498333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.498420 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.498447 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:30:53.511168 (+ 12721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:30:53.511350 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.58.126:45953: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":127,"threads_started":1}
I20260620 14:30:53.512193 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9543
2026-06-20T14:30:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:30:53.550379 4329 mini_oidc.cc:152] Starting JWKS server
I20260620 14:30:53.551234 4329 webserver.cc:460] Webserver started at https://127.0.0.1:37759/ using document root <none> and password file <none>
I20260620 14:30:53.551321 4329 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:30:53.551575 4329 webserver.cc:460] Webserver started at http://127.0.0.1:33063/ using document root <none> and password file <none>
2026-06-20T14:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:53Z Disabled control of system clock
I20260620 14:30:53.560060 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45469
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37759/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:30:53.669809 9646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:53.670099 9646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:53.670140 9646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:53.674173 9646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:53.674249 9646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:53.674264 9646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:53.674280 9646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:53.674294 9646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:30:53.674310 9646 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:30:53.674322 9646 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37759/jwks.json
I20260620 14:30:53.679200 9646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37759/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:53.680281 9646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:53.681627 9646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:53.690305 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:53.690239 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:53.690320 9646 server_base.cc:1034] running on GCE node
W20260620 14:30:53.690223 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:53.691084 9646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:53.692251 9646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:53.693418 9646 hybrid_clock.cc:648] HybridClock initialized: now 1781965853693388 us; error 41 us; skew 500 ppm
I20260620 14:30:53.695377 9646 webserver.cc:460] Webserver started at http://127.4.58.126:34477/ using document root <none> and password file <none>
I20260620 14:30:53.695925 9646 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:53.695984 9646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:53.696148 9646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:53.698241 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "93f0392a48d244648f8a762804d51c5b"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "ffb1d7134f5d49910d5b0f1b282c9c98"
server_key_iv: "d9e5789e629bed99a82c9bfc711ade3d"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.698756 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "93f0392a48d244648f8a762804d51c5b"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "ffb1d7134f5d49910d5b0f1b282c9c98"
server_key_iv: "d9e5789e629bed99a82c9bfc711ade3d"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.702452 9646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:30:53.704933 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:53.706214 9646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:53.706321 9646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "93f0392a48d244648f8a762804d51c5b"
format_stamp: "Formatted at 2026-06-20 14:30:53 on dist-test-slave-gqzk"
server_key: "ffb1d7134f5d49910d5b0f1b282c9c98"
server_key_iv: "d9e5789e629bed99a82c9bfc711ade3d"
server_key_version: "encryptionkey@0"
I20260620 14:30:53.706418 9646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:53.733857 9646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:53.734583 9646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:53.734745 9646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:53.741729 9646 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:45469
I20260620 14:30:53.741751 9712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:45469 every 8 connection(s)
I20260620 14:30:53.742708 9646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:53.745834 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9646
I20260620 14:30:53.745936 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:53.745963 9713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:53.746181 4329 external_mini_cluster.cc:1428] Setting key d59bfd39657763bb277125310206b6b2
I20260620 14:30:53.752575 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b: Bootstrap starting.
I20260620 14:30:53.755129 9713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:53.755606 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.747537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.4.58.126:45469)
0620 14:30:53.748159 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.748176 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.748213 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:53.749033 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.749054 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.749459 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.749738 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:53.750096 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.751063 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.752278 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.752810 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.752937 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.753826 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:53.753831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:53.753844 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:53.754249 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:53.754302 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:53.754323 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:53.754406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:53.754561 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:30:53.754834 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260620 14:30:53.755896 9713 log.cc:824] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:53.758015 9713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b: No bootstrap required, opened a new log
I20260620 14:30:53.758307 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:53.761502 9713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } }
I20260620 14:30:53.761737 9713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:53.761799 9713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f0392a48d244648f8a762804d51c5b, State: Initialized, Role: FOLLOWER
I20260620 14:30:53.762207 9713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } }
I20260620 14:30:53.762334 9713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:53.762406 9713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:53.762507 9713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:53.763433 9713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } }
I20260620 14:30:53.763764 9713 leader_election.cc:302] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f0392a48d244648f8a762804d51c5b; no voters:
I20260620 14:30:53.764062 9713 leader_election.cc:288] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:53.764194 9718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:53.764390 9718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [term 1 LEADER]: Becoming Leader. State: Replica: 93f0392a48d244648f8a762804d51c5b, State: Running, Role: LEADER
I20260620 14:30:53.764668 9718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } }
I20260620 14:30:53.765286 9713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:53.766301 9719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f0392a48d244648f8a762804d51c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } } }
I20260620 14:30:53.766287 9720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f0392a48d244648f8a762804d51c5b. Latest consensus state: current_term: 1 leader_uuid: "93f0392a48d244648f8a762804d51c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0392a48d244648f8a762804d51c5b" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45469 } } }
I20260620 14:30:53.766433 9719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.766448 9720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:53.766848 9727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:53.770114 9727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:53.776324 9727 catalog_manager.cc:1347] Generated new cluster ID: f1f9be593bc64d77ba4d68f2851c48a3
I20260620 14:30:53.776410 9727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:53.791843 9727 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:53.793197 9727 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:53.800590 9727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f0392a48d244648f8a762804d51c5b: Generated new TSK 0
I20260620 14:30:53.801497 9727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:53.825057 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:53.819223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.4.58.126:45469)
0620 14:30:53.819375 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:53.819379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:53.819511 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:30:53.819668 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:53.819671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:53.819732 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:53.819855 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:30:53.820076 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.820954 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.821798 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.822151 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.822250 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.822543 (+ 293us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:30:53.824640 (+ 2097us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:30:53.824912 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 14:30:53.825131 9716 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:30:53.826645 9742 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:53.819105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45469 (local address 127.0.0.1:44916)
0620 14:30:53.819421 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:53.819435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:53.819540 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:53.819864 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:53.819867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:53.819874 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:30:53.819976 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.819981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.821067 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.821070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.821693 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:53.821701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:53.822256 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:53.822258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:53.822318 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:53.822342 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:30:53.824832 (+ 2490us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:30:53.826471 (+ 1639us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.58.126:45469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37759/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1431,"thread_start_us":91,"threads_started":1}
I20260620 14:30:53.827390 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9646
E20260620 14:30:53.837270 9743 webserver.cc:533] Webserver: sslize failed:
E20260620 14:30:53.837350 9743 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:30:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:53Z Disabled control of system clock
I20260620 14:30:53.875270 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42431
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:37477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:42431
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/testchainca.cert with env {}
W20260620 14:30:53.983824 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:53.984097 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:53.984154 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:53.988196 9747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:53.988262 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:53.988284 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:53.988304 9747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:53.988325 9747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:30:53.988355 9747 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:30:53.988379 9747 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:30:53.993685 9747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:42431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:53.994722 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:53.995739 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:54.004472 9752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:54.004432 9753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:54.004621 9747 server_base.cc:1034] running on GCE node
W20260620 14:30:54.004472 9755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:54.005446 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:54.006603 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:54.007776 9747 hybrid_clock.cc:648] HybridClock initialized: now 1781965854007756 us; error 30 us; skew 500 ppm
I20260620 14:30:54.009749 9747 webserver.cc:460] Webserver started at http://127.4.58.126:38037/ using document root <none> and password file <none>
I20260620 14:30:54.010325 9747 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:54.010385 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:54.010545 9747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:54.012604 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "f30776814e5d485fac48c73bc700ee61"
format_stamp: "Formatted at 2026-06-20 14:30:54 on dist-test-slave-gqzk"
server_key: "4346f656cbf84477f50a8a47d6d6a548"
server_key_iv: "ca96b5aa315eb3028d872da502a9595e"
server_key_version: "encryptionkey@0"
I20260620 14:30:54.013162 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "f30776814e5d485fac48c73bc700ee61"
format_stamp: "Formatted at 2026-06-20 14:30:54 on dist-test-slave-gqzk"
server_key: "4346f656cbf84477f50a8a47d6d6a548"
server_key_iv: "ca96b5aa315eb3028d872da502a9595e"
server_key_version: "encryptionkey@0"
I20260620 14:30:54.017061 9747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:30:54.019680 9761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:54.021132 9747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:54.021246 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "f30776814e5d485fac48c73bc700ee61"
format_stamp: "Formatted at 2026-06-20 14:30:54 on dist-test-slave-gqzk"
server_key: "4346f656cbf84477f50a8a47d6d6a548"
server_key_iv: "ca96b5aa315eb3028d872da502a9595e"
server_key_version: "encryptionkey@0"
I20260620 14:30:54.021400 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:54.032603 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:54.033329 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:54.033479 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:54.039898 9747 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:42431
I20260620 14:30:54.039921 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:42431 every 8 connection(s)
I20260620 14:30:54.040860 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:54.044005 9814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:54.049691 9814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: Bootstrap starting.
I20260620 14:30:54.050673 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9747
I20260620 14:30:54.050769 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:54.051010 4329 external_mini_cluster.cc:1428] Setting key 696cdc7ce1d26e5ddf20a06dfcfc8f62
I20260620 14:30:54.052531 9814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:54.053332 9814 log.cc:824] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:54.055372 9814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: No bootstrap required, opened a new log
I20260620 14:30:54.059429 9814 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } }
I20260620 14:30:54.059706 9814 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:54.059777 9814 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30776814e5d485fac48c73bc700ee61, State: Initialized, Role: FOLLOWER
I20260620 14:30:54.060207 9814 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } }
I20260620 14:30:54.060329 9814 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:54.060393 9814 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:54.060487 9814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:54.060693 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:54.052169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.4.58.126:42431)
0620 14:30:54.052953 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:54.052968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:54.053006 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:54.053781 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:54.053803 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:54.054156 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:54.054453 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:54.054787 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.055786 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.057039 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.057603 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.057733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:54.058667 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:54.058677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:54.058690 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:54.059109 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:54.059171 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:54.059193 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:54.059290 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:54.059492 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:30:54.059794 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":126,"threads_started":1}
I20260620 14:30:54.061602 9814 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } }
I20260620 14:30:54.061990 9814 leader_election.cc:302] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30776814e5d485fac48c73bc700ee61; no voters:
I20260620 14:30:54.062295 9814 leader_election.cc:288] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:54.062412 9819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:54.062669 9819 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [term 1 LEADER]: Becoming Leader. State: Replica: f30776814e5d485fac48c73bc700ee61, State: Running, Role: LEADER
I20260620 14:30:54.063251 9819 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } }
I20260620 14:30:54.063545 9814 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:54.063948 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:54.064893 9821 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30776814e5d485fac48c73bc700ee61. Latest consensus state: current_term: 1 leader_uuid: "f30776814e5d485fac48c73bc700ee61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } } }
I20260620 14:30:54.065032 9821 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:54.065330 9820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30776814e5d485fac48c73bc700ee61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30776814e5d485fac48c73bc700ee61" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42431 } } }
I20260620 14:30:54.065428 9820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:54.068017 9834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:54.068080 9834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:54.068185 9830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:54.069193 9830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:54.075374 9830 catalog_manager.cc:1347] Generated new cluster ID: ce8589268f0442e78c01d13dff36d739
I20260620 14:30:54.075448 9830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:54.094664 9830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:54.095670 9830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:54.106316 9830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30776814e5d485fac48c73bc700ee61: Generated new TSK 0
I20260620 14:30:54.107267 9830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:54.194160 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:54.123448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.4.58.126:42431)
0620 14:30:54.123598 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:54.123603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:54.123771 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:30:54.123927 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:54.123930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:54.123995 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:54.124127 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:30:54.124431 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.125361 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.126313 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.126662 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.126761 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:54.127048 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:30:54.193639 (+ 66591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:30:54.194004 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:30:54.194242 9817 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:30:54.194314 9843 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:54.123306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:42431 (local address 127.0.0.1:44560)
0620 14:30:54.123677 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:54.123690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:54.123798 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:54.124127 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:54.124130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:54.124137 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:30:54.124254 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:54.124262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.125489 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.125493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:54.126197 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:54.126212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:54.126766 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:54.126769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:54.126836 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:54.126878 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:30:54.193923 (+ 67045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:30:54.194144 (+ 221us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.58.126:42431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260620 14:30:54.194998 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9747
2026-06-20T14:30:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (345 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:54 dist-test-slave-gqzk krb5kdc[9855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:54 dist-test-slave-gqzk krb5kdc[9855](info): set up 2 sockets
Jun 20 14:30:54 dist-test-slave-gqzk krb5kdc[9855](info): commencing operation
krb5kdc: starting...
W20260620 14:30:55.393002 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:55 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:30:55.513723 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40031
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:55.624895 9868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:55.625196 9868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:55.625253 9868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:55.629329 9868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:55.629405 9868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:55.629432 9868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:55.629462 9868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:55.629484 9868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:55.634964 9868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:55.636137 9868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:55.637298 9868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:55.645905 9873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:55.645877 9876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:55.645865 9874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:55.646124 9868 server_base.cc:1034] running on GCE node
I20260620 14:30:55.646718 9868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:55.647862 9868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:55.649019 9868 hybrid_clock.cc:648] HybridClock initialized: now 1781965855648999 us; error 40 us; skew 500 ppm
Jun 20 14:30:55 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:55.651787 9868 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:30:55.652725 9868 webserver.cc:460] Webserver started at http://127.4.58.126:36297/ using document root <none> and password file <none>
I20260620 14:30:55.653303 9868 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:55.653370 9868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:55.653600 9868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:55.655772 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "f7735acb57044a62ba4b8a33ef90e594"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "6df32f277b857c41d02abcc58f97a6dc"
server_key_iv: "3c8d7d7a5c2087cb243107e946a2756f"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.656334 9868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "f7735acb57044a62ba4b8a33ef90e594"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "6df32f277b857c41d02abcc58f97a6dc"
server_key_iv: "3c8d7d7a5c2087cb243107e946a2756f"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.660269 9868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 14:30:55.662894 9883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:55.664211 9868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:55.664336 9868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "f7735acb57044a62ba4b8a33ef90e594"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "6df32f277b857c41d02abcc58f97a6dc"
server_key_iv: "3c8d7d7a5c2087cb243107e946a2756f"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.664451 9868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:55.701841 9868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:55.705111 9868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:55.705300 9868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:55.712319 9868 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:40031
I20260620 14:30:55.712332 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:40031 every 8 connection(s)
I20260620 14:30:55.713373 9868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:55.716296 9936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:55.720198 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9868
I20260620 14:30:55.720309 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:55.720546 4329 external_mini_cluster.cc:1428] Setting key 47d9050d51af566bfa0096efa5bd8cf6
I20260620 14:30:55.722400 9936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594: Bootstrap starting.
I20260620 14:30:55.724954 9936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:55.725687 9936 log.cc:824] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:55.727815 9936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594: No bootstrap required, opened a new log
Jun 20 14:30:55 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:55.731760 9936 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } }
I20260620 14:30:55.732012 9936 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:55.732070 9936 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7735acb57044a62ba4b8a33ef90e594, State: Initialized, Role: FOLLOWER
I20260620 14:30:55.732571 9936 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } }
I20260620 14:30:55.732733 9936 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:55.732824 9936 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:55.732924 9936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:55.734035 9936 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } }
I20260620 14:30:55.734395 9936 leader_election.cc:302] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7735acb57044a62ba4b8a33ef90e594; no voters:
I20260620 14:30:55.734719 9936 leader_election.cc:288] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:55.734813 9941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:55.734980 9941 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [term 1 LEADER]: Becoming Leader. State: Replica: f7735acb57044a62ba4b8a33ef90e594, State: Running, Role: LEADER
I20260620 14:30:55.735106 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:55.722058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.4.58.126:40031)
0620 14:30:55.722493 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:55.722513 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:55.722556 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:30:55.723229 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:55.723247 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:55.723601 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:55.723900 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:55.724841 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.725902 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.726936 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.727421 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.727542 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:55.730247 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:55.730253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:55.730268 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:55.730303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:55.731885 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:55.732357 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:55.732363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:55.732369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:55.732515 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:55.732825 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:55.732828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:55.732831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:55.733299 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:55.733475 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:55.733774 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:30:55.734066 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260620 14:30:55.735353 9941 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } }
I20260620 14:30:55.736097 9936 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:55.736639 9943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7735acb57044a62ba4b8a33ef90e594. Latest consensus state: current_term: 1 leader_uuid: "f7735acb57044a62ba4b8a33ef90e594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } } }
I20260620 14:30:55.737418 9943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:55.737285 9942 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7735acb57044a62ba4b8a33ef90e594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7735acb57044a62ba4b8a33ef90e594" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40031 } } }
I20260620 14:30:55.737754 9942 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:55.738559 9950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:55.741590 9950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:55.747759 9950 catalog_manager.cc:1347] Generated new cluster ID: 8371ee415669476c9b5327e064f686c3
I20260620 14:30:55.747855 9950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:55.767460 9950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:55.769039 9950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:55.774904 9950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7735acb57044a62ba4b8a33ef90e594: Generated new TSK 0
I20260620 14:30:55.775663 9950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:30:55.803282 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:55.917981 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:55.918268 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:55.918330 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:55.922528 9962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:55.922603 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:55.922685 9962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:55.928100 9962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:55.929414 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:55.930487 9962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:55.940295 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:55.940299 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:55.940299 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:55.940912 9962 server_base.cc:1034] running on GCE node
I20260620 14:30:55.941339 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:55.942026 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:55.943174 9962 hybrid_clock.cc:648] HybridClock initialized: now 1781965855943145 us; error 41 us; skew 500 ppm
Jun 20 14:30:55 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:55.945940 9962 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:30:55.946947 9962 webserver.cc:460] Webserver started at http://127.4.58.65:39443/ using document root <none> and password file <none>
I20260620 14:30:55.947508 9962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:55.947577 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:55.947781 9962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:55.950006 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "badc08f3d4f84178a167788633b107ea"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "0718ba7dff2d5eaf061928e44941a43a"
server_key_iv: "fb4b9b394ee800c454a5d53c979a9e70"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.950551 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "badc08f3d4f84178a167788633b107ea"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "0718ba7dff2d5eaf061928e44941a43a"
server_key_iv: "fb4b9b394ee800c454a5d53c979a9e70"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.954393 9962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:55.956928 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:55.958194 9962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:55.958292 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "badc08f3d4f84178a167788633b107ea"
format_stamp: "Formatted at 2026-06-20 14:30:55 on dist-test-slave-gqzk"
server_key: "0718ba7dff2d5eaf061928e44941a43a"
server_key_iv: "fb4b9b394ee800c454a5d53c979a9e70"
server_key_version: "encryptionkey@0"
I20260620 14:30:55.958382 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:55.972249 9962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:55.975772 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:55.975973 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:55.976557 9962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:55.977425 9962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:55.977473 9962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:55.977525 9962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:55.977555 9962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:55.985952 9962 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:34567
I20260620 14:30:55.985987 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:34567 every 8 connection(s)
I20260620 14:30:55.986850 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:55.990136 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 9962
I20260620 14:30:55.990226 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:55.990473 4329 external_mini_cluster.cc:1428] Setting key 2d329057d50774852c3302ce636b8e10
Jun 20 14:30:55 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965855, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:56.000890 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:55.988636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55175 (local address 127.4.58.126:40031)
0620 14:30:55.988824 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:55.988829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:55.989718 (+ 889us) server_negotiation.cc:407] Connection header received
0620 14:30:55.991153 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:55.991157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:55.991211 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:55.991282 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:55.992476 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.993047 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.994214 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.994446 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.994537 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:55.997176 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:55.997179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:55.997181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:55.997208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:55.998843 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:55.999397 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:55.999400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:55.999401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:55.999485 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:55.999797 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:55.999799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:55.999800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.000002 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.000105 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.000605 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.000718 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:56.001930 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:55.989057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40031 (local address 127.4.58.65:55175)
0620 14:30:55.989567 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:55.989616 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:55.990947 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:55.991468 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:55.991476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:55.991824 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:55.992326 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:55.992340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.993171 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.993173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:55.994092 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:55.994101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:55.994591 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:55.994595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:55.994760 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:55.995308 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:55.995326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:55.996960 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:55.998985 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:55.998991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:55.999006 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:55.999300 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:55.999573 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:55.999576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:55.999577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:55.999726 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:56.000119 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:56.000124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:56.000390 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 14:30:56.000575 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:30:56.001015 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260620 14:30:56.003250 10091 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40031
I20260620 14:30:56.003539 10091 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:56.004184 10091 heartbeater.cc:511] Master 127.4.58.126:40031 requested a full tablet report, sending...
I20260620 14:30:56.006026 9898 ts_manager.cc:194] Registered new tserver with Master: badc08f3d4f84178a167788633b107ea (127.4.58.65:34567)
I20260620 14:30:56.007241 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:55175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:30:56.019259 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:56.130071 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:56.130366 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:56.130409 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:56.134593 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:56.134658 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:56.134718 10096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:56.140229 10096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:56.141328 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:56.142292 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:56.151849 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:56.151849 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:56.151852 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:56.151892 10096 server_base.cc:1034] running on GCE node
I20260620 14:30:56.152535 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:56.153231 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:56.154443 10096 hybrid_clock.cc:648] HybridClock initialized: now 1781965856154431 us; error 69 us; skew 500 ppm
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.157407 10096 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:30:56.158352 10096 webserver.cc:460] Webserver started at http://127.4.58.66:40633/ using document root <none> and password file <none>
I20260620 14:30:56.158923 10096 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:56.158993 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:56.159201 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:56.161340 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "209cfc6b7475441fb75a131515793825"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "1ab8cdea8b5204a8f7216780c82a4a8f"
server_key_iv: "acff3c112528b835ddd4546e80aa2ccb"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.161895 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "209cfc6b7475441fb75a131515793825"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "1ab8cdea8b5204a8f7216780c82a4a8f"
server_key_iv: "acff3c112528b835ddd4546e80aa2ccb"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.165756 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:30:56.168277 10111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.169687 10096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:30:56.169797 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "209cfc6b7475441fb75a131515793825"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "1ab8cdea8b5204a8f7216780c82a4a8f"
server_key_iv: "acff3c112528b835ddd4546e80aa2ccb"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.169900 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:56.191074 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:56.194206 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:56.194384 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:56.194974 10096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:56.195830 10096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:56.195878 10096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.195931 10096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:56.195961 10096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.204533 10096 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:45979
I20260620 14:30:56.204560 10224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:45979 every 8 connection(s)
I20260620 14:30:56.205459 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:56.205765 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10096
I20260620 14:30:56.205873 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:56.206198 4329 external_mini_cluster.cc:1428] Setting key 3092e7c0a1782e82dd0b4daae20060a5
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:56.222002 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.207182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:58275 (local address 127.4.58.126:40031)
0620 14:30:56.207310 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.207314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.208350 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 14:30:56.209963 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.209966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.210021 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.210092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.211301 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.212116 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.213378 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.213724 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.213843 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.218529 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:56.218533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:56.218534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:56.218561 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:56.219912 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.220424 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.220433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.220434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.220508 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.220903 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.220906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.220907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.221093 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.221178 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.221700 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.221832 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:56.222913 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.207669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40031 (local address 127.4.58.66:58275)
0620 14:30:56.208175 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:56.208230 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:56.209751 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:56.210277 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:56.210285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:56.210647 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:56.211128 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:56.211142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.212248 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.212251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:56.213257 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:56.213267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.213878 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.213881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:56.214036 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.214581 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:56.214599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:56.218334 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:56.220032 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:56.220038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:56.220058 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:56.220319 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:56.220590 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:56.220593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:56.220595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:56.220805 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:56.221189 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:56.221196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:56.221462 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 14:30:56.221629 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:30:56.221989 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260620 14:30:56.224102 10225 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40031
I20260620 14:30:56.224326 10225 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:56.224871 10225 heartbeater.cc:511] Master 127.4.58.126:40031 requested a full tablet report, sending...
I20260620 14:30:56.225895 9898 ts_manager.cc:194] Registered new tserver with Master: 209cfc6b7475441fb75a131515793825 (127.4.58.66:45979)
I20260620 14:30:56.226641 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:58275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:30:56.233438 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:56.343653 10230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:56.343940 10230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:56.343982 10230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:56.348127 10230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:56.348188 10230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:56.348249 10230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:56.353665 10230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:56.354750 10230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:56.355762 10230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:56.366025 10230 server_base.cc:1034] running on GCE node
W20260620 14:30:56.365965 10235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:56.366015 10236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:56.365971 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:56.366698 10230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:56.367300 10230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:56.368497 10230 hybrid_clock.cc:648] HybridClock initialized: now 1781965856368476 us; error 35 us; skew 500 ppm
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.371410 10230 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:30:56.372289 10230 webserver.cc:460] Webserver started at http://127.4.58.67:40887/ using document root <none> and password file <none>
I20260620 14:30:56.372874 10230 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:56.372957 10230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:56.373162 10230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:56.375300 10230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "5f100e481b9d4a029210020a89bcf1cc"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "4ffabe8cdb25e8069cf7bd824702a5d3"
server_key_iv: "2c1ce219aee9c2d18208550f7fd931e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.375872 10230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "5f100e481b9d4a029210020a89bcf1cc"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "4ffabe8cdb25e8069cf7bd824702a5d3"
server_key_iv: "2c1ce219aee9c2d18208550f7fd931e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.379580 10230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:30:56.382050 10245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.383311 10230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:56.383427 10230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "5f100e481b9d4a029210020a89bcf1cc"
format_stamp: "Formatted at 2026-06-20 14:30:56 on dist-test-slave-gqzk"
server_key: "4ffabe8cdb25e8069cf7bd824702a5d3"
server_key_iv: "2c1ce219aee9c2d18208550f7fd931e4"
server_key_version: "encryptionkey@0"
I20260620 14:30:56.383535 10230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:56.399446 10230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:56.402748 10230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:56.402941 10230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:56.403532 10230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:56.404346 10230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:56.404392 10230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.404443 10230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:56.404471 10230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:56.412739 10230 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:45867
I20260620 14:30:56.412806 10358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:45867 every 8 connection(s)
I20260620 14:30:56.413686 10230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:56.419863 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10230
I20260620 14:30:56.419987 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:56.420267 4329 external_mini_cluster.cc:1428] Setting key 65d094a6f10fc22cb6dd97a86d288ff9
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:30:56.427704 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.415564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46951 (local address 127.4.58.126:40031)
0620 14:30:56.415728 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.415731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.416605 (+ 874us) server_negotiation.cc:407] Connection header received
0620 14:30:56.418046 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.418049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.418110 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.418186 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.419419 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.420127 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.421687 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.421904 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.421979 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.424412 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:56.424415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:56.424417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:56.424449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:56.425717 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.426193 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.426196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.426197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.426262 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.426629 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.426632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.426633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.426843 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.426933 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.427466 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.427578 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:30:56.428570 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.415943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40031 (local address 127.4.58.67:46951)
0620 14:30:56.416433 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:56.416479 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:56.417786 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:56.418369 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:56.418377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:56.418747 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:56.419255 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:56.419269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.420364 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.420369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:56.421552 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:56.421562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.422050 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.422054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:56.422261 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.422816 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:56.422833 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:56.424248 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:56.425821 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:56.425827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:56.425840 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:56.426116 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:56.426365 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:56.426368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:56.426370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:56.426519 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:56.426968 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:56.426975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:56.427229 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 14:30:56.427405 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:30:56.427682 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260620 14:30:56.429845 10359 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40031
I20260620 14:30:56.430116 10359 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:56.430610 10359 heartbeater.cc:511] Master 127.4.58.126:40031 requested a full tablet report, sending...
I20260620 14:30:56.431702 9898 ts_manager.cc:194] Registered new tserver with Master: 5f100e481b9d4a029210020a89bcf1cc (127.4.58.67:45867)
I20260620 14:30:56.432265 9898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:46951
I20260620 14:30:56.434085 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/test-admin.keytab.
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.449853 4329 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:30:56.452593 10366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:56.451242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.4.58.65:34567)
0620 14:30:56.451557 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.451566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.451591 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:56.451643 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.451648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.451830 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.451960 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.452398 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38010: BlockingRecv error: recv got EOF from 127.0.0.1:38010 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
W20260620 14:30:56.453001 4329 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.454031 4329 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.454869 4329 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:56.464447 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.455750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.4.58.65:34567)
0620 14:30:56.455876 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.455879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.455951 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:30:56.456057 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.456059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.456097 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.456160 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.456683 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.457567 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.458386 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.458779 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.458863 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.461127 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:56.461137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:56.461172 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:56.461197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:56.462667 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.463120 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.463125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.463127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.463209 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.463466 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.463469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.463471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.463730 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.463932 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.464134 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.464282 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:30:56.468222 10366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:56.467214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.4.58.65:34567)
0620 14:30:56.467337 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.467341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.467433 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:30:56.467573 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.467576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.467618 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.467680 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.468111 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38024: BlockingRecv error: recv got EOF from 127.0.0.1:38024 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:30:56.468357 4329 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.469425 4329 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.470273 4329 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:56.478416 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.471109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.4.58.65:34567)
0620 14:30:56.471225 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.471229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.471292 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:30:56.471424 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.471427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.471477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.471540 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.472021 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.472553 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.473422 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.473665 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.473753 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.475804 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:56.475808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:56.475811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:56.475836 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:56.476967 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.477338 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.477342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.477344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.477416 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.477656 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.477660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.477661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.477823 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.477931 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.478160 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.478264 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:30:56.480952 10366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:56.480080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.4.58.65:34567)
0620 14:30:56.480192 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.480196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.480274 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:30:56.480383 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.480385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.480421 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.480481 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.480833 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38038: BlockingRecv error: recv got EOF from 127.0.0.1:38038 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:30:56.481029 4329 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965824702348-4329-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.481990 4329 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:56.482820 4329 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[9855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:30:56.491360 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:56.483692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.4.58.65:34567)
0620 14:30:56.483806 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:56.483810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:56.483881 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:30:56.484025 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:56.484028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:56.484069 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:56.484131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:56.484614 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.485160 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.485906 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:56.486122 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:56.486206 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:56.488550 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:56.488553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:56.488555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:56.488579 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:56.489780 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.490170 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.490174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.490175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.490275 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:56.490600 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:56.490603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:56.490604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:56.490787 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:56.490899 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:56.491139 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:30:56.491241 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:30:56.492486 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9962
I20260620 14:30:56.498063 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10096
I20260620 14:30:56.502774 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10230
I20260620 14:30:56.507261 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 9868
2026-06-20T14:30:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2072 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[10379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[10379](info): set up 2 sockets
Jun 20 14:30:56 dist-test-slave-gqzk krb5kdc[10379](info): commencing operation
krb5kdc: starting...
W20260620 14:30:57.468876 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:30:57 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab.
Entry for principal oryx/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab.
I20260620 14:30:57.591693 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46443
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab
--principal=oryx/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:57.701545 10392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:57.701839 10392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:57.701895 10392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:57.705926 10392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:57.706003 10392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:57.706030 10392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:57.706050 10392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:57.706070 10392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:57.711580 10392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab
--principal=oryx/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:57.712671 10392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:57.713732 10392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:57.722664 10397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:57.722725 10392 server_base.cc:1034] running on GCE node
W20260620 14:30:57.722680 10400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:57.722676 10398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:57.723443 10392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:57.724601 10392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:57.725780 10392 hybrid_clock.cc:648] HybridClock initialized: now 1781965857725761 us; error 35 us; skew 500 ppm
Jun 20 14:30:57 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:57.728576 10392 init.cc:377] Logged in from keytab as oryx/127.4.58.126@KRBTEST.COM (short username oryx)
I20260620 14:30:57.729561 10392 webserver.cc:460] Webserver started at http://127.4.58.126:38185/ using document root <none> and password file <none>
I20260620 14:30:57.730120 10392 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:57.730188 10392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:57.730374 10392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:57.732465 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "8bd903a427b648c59251747d73ab05d3"
format_stamp: "Formatted at 2026-06-20 14:30:57 on dist-test-slave-gqzk"
server_key: "81b2930277249dea67465b21e8e5fa99"
server_key_iv: "885c1e236bdd9e4ebb48c6857ce0438e"
server_key_version: "encryptionkey@0"
I20260620 14:30:57.733021 10392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "8bd903a427b648c59251747d73ab05d3"
format_stamp: "Formatted at 2026-06-20 14:30:57 on dist-test-slave-gqzk"
server_key: "81b2930277249dea67465b21e8e5fa99"
server_key_iv: "885c1e236bdd9e4ebb48c6857ce0438e"
server_key_version: "encryptionkey@0"
I20260620 14:30:57.736835 10392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:30:57.739420 10407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:57.740664 10392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:57.740846 10392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "8bd903a427b648c59251747d73ab05d3"
format_stamp: "Formatted at 2026-06-20 14:30:57 on dist-test-slave-gqzk"
server_key: "81b2930277249dea67465b21e8e5fa99"
server_key_iv: "885c1e236bdd9e4ebb48c6857ce0438e"
server_key_version: "encryptionkey@0"
I20260620 14:30:57.740967 10392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:57.758708 10392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:57.762199 10392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:57.762406 10392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:57.769622 10392 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:46443
I20260620 14:30:57.769646 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:46443 every 8 connection(s)
I20260620 14:30:57.770637 10392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:57.773751 10460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:57.778309 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10392
I20260620 14:30:57.778446 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:30:57.778714 4329 external_mini_cluster.cc:1428] Setting key ab98b9285d0eb7c04d6c710bc2cfd0b3
I20260620 14:30:57.780256 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Bootstrap starting.
I20260620 14:30:57.782912 10460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:57.783727 10460 log.cc:824] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:57.785804 10460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: No bootstrap required, opened a new log
Jun 20 14:30:57 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:57.790134 10460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } }
I20260620 14:30:57.790452 10460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:57.790509 10460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd903a427b648c59251747d73ab05d3, State: Initialized, Role: FOLLOWER
I20260620 14:30:57.791038 10460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } }
I20260620 14:30:57.791183 10460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:57.791256 10460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:57.791360 10460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:57.792533 10460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } }
I20260620 14:30:57.792944 10460 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd903a427b648c59251747d73ab05d3; no voters:
I20260620 14:30:57.793345 10460 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:57.793493 10465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:57.793720 10465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd903a427b648c59251747d73ab05d3, State: Running, Role: LEADER
I20260620 14:30:57.794083 10465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } }
I20260620 14:30:57.794178 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:57.780210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.4.58.126:46443)
0620 14:30:57.780673 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:57.780690 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:57.780728 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:57.781458 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:57.781477 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:57.781851 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:57.782175 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:57.783130 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:57.784256 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:57.785205 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:57.785711 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:57.785869 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:57.788550 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:57.788558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:57.788582 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:57.788620 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:57.790702 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:57.791213 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:57.791220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:57.791226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:57.791372 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:57.791730 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:57.791734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:57.791735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:57.792287 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:57.792493 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:57.792703 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:30:57.793073 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260620 14:30:57.794677 10460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:57.797209 10467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } }
I20260620 14:30:57.797350 10467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:57.797387 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd903a427b648c59251747d73ab05d3. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } }
I20260620 14:30:57.797490 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:57.800036 10479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:57.800117 10479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:57.800179 10481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:57.801038 10481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:57.806816 10481 catalog_manager.cc:1347] Generated new cluster ID: fe63a31684494869a3b4facc8e0746b9
I20260620 14:30:57.806905 10481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:57.814051 10481 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:57.815016 10481 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:57.822535 10481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Generated new TSK 0
I20260620 14:30:57.823436 10481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
I20260620 14:30:57.860076 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:57.975932 10486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:57.976205 10486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:57.976250 10486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:57.980369 10486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:57.980443 10486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:57.980504 10486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:57.986003 10486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:57.987115 10486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:57.988144 10486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:57.998221 10486 server_base.cc:1034] running on GCE node
W20260620 14:30:57.998170 10491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:57.998184 10492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:57.998148 10494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:57.998878 10486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:57.999524 10486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.000689 10486 hybrid_clock.cc:648] HybridClock initialized: now 1781965858000656 us; error 43 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.008615 10486 init.cc:377] Logged in from keytab as oryx/127.4.58.65@KRBTEST.COM (short username oryx)
I20260620 14:30:58.009686 10486 webserver.cc:460] Webserver started at http://127.4.58.65:44113/ using document root <none> and password file <none>
I20260620 14:30:58.010334 10486 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.010411 10486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.010618 10486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:58.012912 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "b57256c51e4e73b4291844b19da347ad"
server_key_iv: "d8b56b4c6025b2600aecaebc08805c8b"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.013481 10486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "b57256c51e4e73b4291844b19da347ad"
server_key_iv: "d8b56b4c6025b2600aecaebc08805c8b"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.017320 10486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:30:58.020074 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.021497 10486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:58.021620 10486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "b57256c51e4e73b4291844b19da347ad"
server_key_iv: "d8b56b4c6025b2600aecaebc08805c8b"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.021734 10486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.035259 10486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:58.038661 10486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:58.038851 10486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:58.039436 10486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:58.040271 10486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:58.040316 10486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.040355 10486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:58.040378 10486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.049319 10486 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:36975
I20260620 14:30:58.049340 10614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:36975 every 8 connection(s)
I20260620 14:30:58.050264 10486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:58.056864 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10486
I20260620 14:30:58.056996 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:30:58.057272 4329 external_mini_cluster.cc:1428] Setting key 9f587cef3464599e03326e9bb7896d87
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:58.064446 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.052205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:40947 (local address 127.4.58.126:46443)
0620 14:30:58.052339 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.052343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.053275 (+ 932us) server_negotiation.cc:407] Connection header received
0620 14:30:58.054619 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.054622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.054679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.054756 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.056122 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.056695 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.058058 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.058289 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.058394 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.060866 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.060869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.060871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.060897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.062228 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.062880 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.062884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.062886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.062981 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.063291 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.063294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.063296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.063549 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.063655 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.064136 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.064270 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:30:58.065289 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.052589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:40947)
0620 14:30:58.053115 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.053161 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.054400 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.054956 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.054963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.055339 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.055907 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.055921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.056834 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.056838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.057918 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.057929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.058438 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.058443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.058618 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.059216 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.059233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.060667 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.062364 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.062371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.062391 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.062762 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.063063 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.063065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.063066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.063207 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.063644 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.063649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.063899 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.064067 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.064335 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":171,"threads_started":1}
WARNING: no policy specified for oryx/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:30:58.066773 10615 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:58.067035 10615 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:58.067559 10615 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.66@KRBTEST.COM" created.
I20260620 14:30:58.069254 10424 ts_manager.cc:194] Registered new tserver with Master: 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975)
I20260620 14:30:58.070708 10424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:40947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
I20260620 14:30:58.084450 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:58.192914 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:58.193250 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:58.193310 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:58.197695 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:58.197769 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:58.197856 10620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:58.203351 10620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:58.204439 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:58.205513 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:58.215049 10625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.215240 10620 server_base.cc:1034] running on GCE node
W20260620 14:30:58.215137 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.215055 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.215758 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:58.216427 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.217577 10620 hybrid_clock.cc:648] HybridClock initialized: now 1781965858217562 us; error 42 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.220273 10620 init.cc:377] Logged in from keytab as oryx/127.4.58.66@KRBTEST.COM (short username oryx)
I20260620 14:30:58.221226 10620 webserver.cc:460] Webserver started at http://127.4.58.66:38177/ using document root <none> and password file <none>
I20260620 14:30:58.221776 10620 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.221837 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.222035 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:58.224089 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "d75c336580736bd352051b0cc673cc7a"
server_key_iv: "ad8c2a020acca03dda6af63e9735f6a9"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.224632 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "d75c336580736bd352051b0cc673cc7a"
server_key_iv: "ad8c2a020acca03dda6af63e9735f6a9"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.228438 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 14:30:58.230914 10635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.232197 10620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:58.232306 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "d75c336580736bd352051b0cc673cc7a"
server_key_iv: "ad8c2a020acca03dda6af63e9735f6a9"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.232414 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.248108 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:58.251009 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:58.251188 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:58.251786 10620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:58.252609 10620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:58.252655 10620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.252697 10620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:58.252727 10620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.261119 10620 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:40625
I20260620 14:30:58.261179 10748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:40625 every 8 connection(s)
I20260620 14:30:58.262039 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:58.270825 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10620
I20260620 14:30:58.270936 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:30:58.271219 4329 external_mini_cluster.cc:1428] Setting key fd76194faa5941f9782f3126ec59e650
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:58.276036 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.263869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:35277 (local address 127.4.58.126:46443)
0620 14:30:58.264027 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.264032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.265020 (+ 988us) server_negotiation.cc:407] Connection header received
0620 14:30:58.266312 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.266315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.266366 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.266443 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.267662 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.268200 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.269404 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.269655 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.269749 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.272454 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.272465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.272468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.272493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.274017 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.274529 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.274533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.274535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.274622 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.274949 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.274952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.274954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.275148 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.275234 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.275798 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.275910 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:30:58.277262 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.264314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:35277)
0620 14:30:58.264850 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.264911 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.266087 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.266606 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.266613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.267001 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.267514 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.267529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.268301 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.268304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.269260 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.269269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.269846 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.269850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.270007 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.270630 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.270646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.272283 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.274132 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.274138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.274153 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.274423 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.274704 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.274707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.274709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.274860 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.275243 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.275250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.275541 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.275752 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.276134 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260620 14:30:58.278414 10749 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:58.278688 10749 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:58.279300 10749 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
I20260620 14:30:58.280400 10424 ts_manager.cc:194] Registered new tserver with Master: c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
WARNING: no policy specified for oryx/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:30:58.281065 10424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:35277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
I20260620 14:30:58.299980 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:58.412963 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:58.413268 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:58.413318 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:58.417809 10754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:58.417889 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:58.417958 10754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:58.423436 10754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:58.424580 10754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:58.425791 10754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:58.435925 10759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.435941 10760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.435925 10762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.436178 10754 server_base.cc:1034] running on GCE node
I20260620 14:30:58.436820 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:58.437551 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.438721 10754 hybrid_clock.cc:648] HybridClock initialized: now 1781965858438699 us; error 40 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.441526 10754 init.cc:377] Logged in from keytab as oryx/127.4.58.67@KRBTEST.COM (short username oryx)
I20260620 14:30:58.442456 10754 webserver.cc:460] Webserver started at http://127.4.58.67:39127/ using document root <none> and password file <none>
I20260620 14:30:58.443013 10754 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.443082 10754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.443285 10754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:58.445564 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "4e201563e402938f868e1b0d8ea9ccaf"
server_key_iv: "f2de99f356141baf30421b88ac30cba0"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.446110 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "4e201563e402938f868e1b0d8ea9ccaf"
server_key_iv: "f2de99f356141baf30421b88ac30cba0"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.449939 10754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:30:58.452420 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.453747 10754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:58.453871 10754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "4e201563e402938f868e1b0d8ea9ccaf"
server_key_iv: "f2de99f356141baf30421b88ac30cba0"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.453990 10754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.483206 10754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:58.486572 10754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:58.486763 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:58.487381 10754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:58.488231 10754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:58.488279 10754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.488332 10754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:58.488360 10754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.497272 10754 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34777
I20260620 14:30:58.497292 10882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34777 every 8 connection(s)
I20260620 14:30:58.498198 10754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:58.506829 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10754
I20260620 14:30:58.506945 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:30:58.507216 4329 external_mini_cluster.cc:1428] Setting key 640a3f49ce28b9a5aca43127a483e685
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:58.511572 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.499988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:33667 (local address 127.4.58.126:46443)
0620 14:30:58.500125 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.500130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.501072 (+ 942us) server_negotiation.cc:407] Connection header received
0620 14:30:58.502409 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.502412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.502479 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.502560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.503765 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.504299 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.505475 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.505691 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.505782 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.508293 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.508297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.508300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.508330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.509859 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.510307 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.510310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.510311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.510377 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.510646 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.510648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.510649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.510809 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.510877 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.511312 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.511414 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:30:58.512545 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.500405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:33667)
0620 14:30:58.500906 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.500967 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.502187 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.502746 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.502753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.503131 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.503616 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.503629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.504421 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.504425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.505343 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.505351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.505831 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.505835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.506002 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.506614 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.506630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.508134 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.509964 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.509970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.509982 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.510233 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.510444 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.510446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.510448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.510580 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.510878 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.510882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.511127 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.511281 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.511602 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":134,"threads_started":1}
I20260620 14:30:58.513753 10883 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:58.514015 10883 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:58.514503 10883 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
I20260620 14:30:58.515513 10424 ts_manager.cc:194] Registered new tserver with Master: d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777)
I20260620 14:30:58.516047 10424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:33667
I20260620 14:30:58.521080 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.58.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.125.keytab.
Entry for principal oryx/127.4.58.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.125.keytab.
I20260620 14:30:58.547500 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:34289
--webserver_interface=127.4.58.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.125.keytab
--principal=oryx/127.4.58.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:46443,127.4.58.125:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:58.660337 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:58.660640 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:58.660679 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:58.664888 10888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:58.664956 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:58.664974 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:58.664989 10888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:58.665004 10888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:58.670681 10888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:46443,127.4.58.125:34289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.125.keytab
--principal=oryx/127.4.58.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.58.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
W20260620 14:30:58.670921 10888 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.58.126:46443,127.4.58.125:34289'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:30:58.671831 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:58.672881 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:58.681284 10893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.681315 10896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.681397 10888 server_base.cc:1034] running on GCE node
W20260620 14:30:58.681241 10894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.682161 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:58.683336 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.684541 10888 hybrid_clock.cc:648] HybridClock initialized: now 1781965858684520 us; error 33 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.687876 10888 init.cc:377] Logged in from keytab as oryx/127.4.58.125@KRBTEST.COM (short username oryx)
I20260620 14:30:58.689011 10888 webserver.cc:460] Webserver started at http://127.4.58.125:35401/ using document root <none> and password file <none>
I20260620 14:30:58.689626 10888 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.689705 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.689951 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:58.692160 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/instance:
uuid: "3c362b2ae7f64faaa35124ea3079e384"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "cdd04f12290cfc9f2895994704c5dcb4"
server_key_iv: "c7d60d28815b7d7c0af4d96aac003037"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.692814 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal/instance:
uuid: "3c362b2ae7f64faaa35124ea3079e384"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "cdd04f12290cfc9f2895994704c5dcb4"
server_key_iv: "c7d60d28815b7d7c0af4d96aac003037"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.697247 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:30:58.699999 10903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.701444 10888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:58.701578 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "3c362b2ae7f64faaa35124ea3079e384"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "cdd04f12290cfc9f2895994704c5dcb4"
server_key_iv: "c7d60d28815b7d7c0af4d96aac003037"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.701689 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.719046 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:58.722213 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:58.722409 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:58.729341 10888 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.125:34289
I20260620 14:30:58.729605 10955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.125:34289 every 8 connection(s)
I20260620 14:30:58.730356 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:58.733317 10956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:58.734143 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10888
I20260620 14:30:58.737771 10956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.125@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:58.749217 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.739863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.4.58.126:46443)
0620 14:30:58.740029 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.740034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.740338 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:30:58.740461 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.740464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.740508 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.740571 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.741387 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.742039 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.743114 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.743461 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.743589 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.745897 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.745901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.745903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.745928 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.747264 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.747757 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.747759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.747760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.747829 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.748100 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.748103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.748105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.748381 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.748469 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.748930 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.749068 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:30:58.750010 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.735741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.4.58.125:34289)
0620 14:30:58.736229 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.736248 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.736295 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:30:58.737150 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.737174 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.737611 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.737993 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.738744 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.740039 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.741066 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.741717 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.741865 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.744924 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.744930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.744942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.744963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.746566 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.747053 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.747058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.747063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.747167 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.747494 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.747497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.747499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.748010 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.748202 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.748530 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.748889 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":327,"thread_start_us":150,"threads_started":1}
I20260620 14:30:58.750010 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.739771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54096)
0620 14:30:58.740227 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.740267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.740375 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.740638 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.740644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.741061 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.741264 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.741271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.742195 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.742197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.742977 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.742987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.743611 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.743613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.743691 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.744241 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.744262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.745756 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.747381 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.747388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.747393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.747666 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.747905 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.747908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.747910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.748012 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.748485 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.748491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.748611 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.748852 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.749031 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":44,"thread_start_us":200,"threads_started":1}
I20260620 14:30:58.751401 10956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.125@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:58.758911 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.751889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36180)
0620 14:30:58.751990 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.752003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.752075 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.752285 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.752287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.752425 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.752516 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.752521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.753573 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.753575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.754078 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.754086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.754508 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.754511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.754595 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.754998 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.755008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.756193 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.757699 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.757702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.757703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.757876 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.758106 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.758109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.758111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.758183 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.758511 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.758513 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.758585 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.758655 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.758760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:30:58.758934 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.752033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.4.58.125:34289)
0620 14:30:58.752120 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.752124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.752135 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:58.752162 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.752165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.752197 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.752267 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.752608 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.753473 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.754166 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.754388 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.754468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.756304 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.756307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.756309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.756329 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.757606 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.757961 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.757963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.757964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.758029 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.758263 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.758266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.758268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.758430 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.758508 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.758698 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.758812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":36}
I20260620 14:30:58.761746 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Bootstrap starting.
I20260620 14:30:58.764029 10956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:58.764744 10956 log.cc:824] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:58.766707 10956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: No bootstrap required, opened a new log
I20260620 14:30:58.769829 10956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } }
I20260620 14:30:58.770051 10956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:58.770108 10956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c362b2ae7f64faaa35124ea3079e384, State: Initialized, Role: FOLLOWER
I20260620 14:30:58.770555 10956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } }
I20260620 14:30:58.771034 10962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } } }
I20260620 14:30:58.771160 10962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:58.771859 10956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:30:58.776515 10973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:58.776602 10973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:58.778656 10888 master_runner.cc:428] Detected that this master 3c362b2ae7f64faaa35124ea3079e384 is joining an existing cluster
I20260620 14:30:58.778725 10888 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.58.125:34289
I20260620 14:30:58.789342 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.781276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36184)
0620 14:30:58.781915 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.781927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.782030 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.782308 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.782311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.782554 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.782677 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.782684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.783646 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.783649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.784529 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.784538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.785079 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.785083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.785239 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.785805 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.785820 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.786335 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.787981 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.787984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.787986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.788157 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.788385 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.788387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.788390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.788481 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.788919 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.788922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.789021 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.789100 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.789197 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":43,"thread_start_us":90,"threads_started":1}
I20260620 14:30:58.789441 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.781332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.4.58.125:34289)
0620 14:30:58.781420 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.781424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.781983 (+ 559us) server_negotiation.cc:407] Connection header received
0620 14:30:58.782112 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.782116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.782149 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.782192 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.782782 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.783524 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.784650 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.784960 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.785058 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.786493 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.786497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.786499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.786521 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.787879 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.788239 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.788242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.788244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.788312 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.788576 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.788579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.788581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.788824 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.788919 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.789157 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.789263 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:30:58.791026 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.780891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54108)
0620 14:30:58.781588 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.781603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.781707 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.782028 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.782031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.782265 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.782394 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.782400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.783474 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.783477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.784358 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.784367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.785079 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.785083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.785193 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.786076 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.786088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.786457 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.788395 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.788397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.788399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.788629 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.789590 (+ 961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.789593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.789595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.789676 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.790532 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.790535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.790625 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.790712 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.790849 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260620 14:30:58.791045 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.780924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.4.58.126:46443)
0620 14:30:58.781042 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.781046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.781677 (+ 631us) server_negotiation.cc:407] Connection header received
0620 14:30:58.781785 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.781788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.781833 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.781908 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.782516 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.783349 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.784487 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.784825 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.784897 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.786593 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.786597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.786599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.786627 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.788143 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.788711 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.788715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.788717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.788825 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.789769 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.789774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.789775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.789995 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.790093 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.790789 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.790903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.126@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:58.805339 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.795898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36194)
0620 14:30:58.796266 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.796317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.796531 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.796859 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.796864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.797236 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.797344 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.797350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.798311 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.798313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.798977 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.798986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.799588 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.799590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.799670 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.800187 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.800203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.801674 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.803519 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.803528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.803535 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.803835 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.804114 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.804117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.804119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.804209 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.804596 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.804603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.804735 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.804916 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.805130 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260620 14:30:58.805847 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.795899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.4.58.125:34289)
0620 14:30:58.796013 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.796018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.796389 (+ 371us) server_negotiation.cc:407] Connection header received
0620 14:30:58.796671 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.796674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.796726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.796833 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.797445 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.798221 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.799079 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.799492 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.799590 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.801803 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.801807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.801810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.801836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.803398 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.803937 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.803941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.803943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.804027 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.804288 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.804291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.804292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.804501 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.804585 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.805582 (+ 997us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.805703 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:30:58.807593 10424 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.58.125:34289: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384"
member_type: NON_VOTER
last_known_addr {
host: "127.4.58.125"
port: 34289
}
attrs {
promote: true
}
}
dest_uuid: "8bd903a427b648c59251747d73ab05d3"
cas_config_opid_index: -1
I20260620 14:30:58.808291 10424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } }
I20260620 14:30:58.810163 10984 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } }
I20260620 14:30:58.811057 10930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:58.811951 10984 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.58.125:34289
I20260620 14:30:58.812203 10930 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:30:58.812340 10983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } }
I20260620 14:30:58.812582 10983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:58.813067 10888 master.cc:537] Master@127.4.58.125:34289 shutting down...
I20260620 14:30:58.813068 10983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.4.58.124@KRBTEST.COM; defaulting to no policy
W20260620 14:30:58.815063 10408 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:30:58.817162 10408 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 -> Peer 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125:34289): Couldn't send request to peer 3c362b2ae7f64faaa35124ea3079e384. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:30:58.817708 10888 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:58.817979 10888 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:58.818060 10888 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.124.keytab.
Entry for principal oryx/127.4.58.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.124.keytab.
I20260620 14:30:58.835611 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:45427
--webserver_interface=127.4.58.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.124.keytab
--principal=oryx/127.4.58.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:58.835685 10888 master.cc:552] Master@127.4.58.125:34289 shutdown complete.
I20260620 14:30:58.835733 10888 master_runner.cc:305] Clearing existing system tablet
I20260620 14:30:58.835976 10888 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.836062 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.837426 10888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:58.838274 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.838531 10888 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.838652 10888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:58.838714 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "3c362b2ae7f64faaa35124ea3079e384"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "cdd04f12290cfc9f2895994704c5dcb4"
server_key_iv: "c7d60d28815b7d7c0af4d96aac003037"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.840934 10888 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:58.843246 10888 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:30:58.843336 10888 log.cc:1196] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:30:58.843636 10888 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Deleting consensus metadata
I20260620 14:30:58.843793 10888 master_runner.cc:315] Copying system tablet from 127.4.58.126:46443
I20260620 14:30:58.844998 10888 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: tablet copy: Beginning tablet copy session from remote peer at address 127.4.58.126:46443
I20260620 14:30:58.853827 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.845896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.4.58.126:46443)
0620 14:30:58.846023 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:58.846027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:58.846064 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:58.846115 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:58.846118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:58.846166 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:58.846245 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:58.846740 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.847582 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.848787 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.849137 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.849240 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.850496 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:58.850500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:58.850503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:58.850532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:58.852106 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.852576 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.852579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.852581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.852660 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:58.852961 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:58.852964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:58.852966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:58.853182 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:58.853276 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:58.853542 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 14:30:58.853658 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:30:58.853842 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:58.845604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54114)
0620 14:30:58.845864 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:58.845879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:58.845995 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:58.846283 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:58.846286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:58.846499 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:58.846631 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.846637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.847698 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.847701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.848638 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:58.848650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:58.849256 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:58.849260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:58.849373 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:58.849899 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:58.849915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:58.850328 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:58.852214 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.852218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.852220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.852481 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.852742 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:58.852745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:58.852785 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:58.852875 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:58.853270 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:58.853273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:58.853387 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:30:58.853491 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:30:58.853636 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260620 14:30:58.854722 10444 tablet_copy_service.cc:140] P 8bd903a427b648c59251747d73ab05d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c362b2ae7f64faaa35124ea3079e384 ({username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:54114)
I20260620 14:30:58.854909 10444 tablet_copy_service.cc:161] P 8bd903a427b648c59251747d73ab05d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c362b2ae7f64faaa35124ea3079e384 at {username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:54114: session id = 3c362b2ae7f64faaa35124ea3079e384-00000000000000000000000000000000
I20260620 14:30:58.856727 10444 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:30:58.858682 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:30:58.861158 10888 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: tablet copy: Starting download of 0 data blocks...
I20260620 14:30:58.861346 10888 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: tablet copy: Starting download of 1 WAL segments...
I20260620 14:30:58.863134 10888 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:30:58.865078 10444 tablet_copy_service.cc:342] P 8bd903a427b648c59251747d73ab05d3: Request end of tablet copy session 3c362b2ae7f64faaa35124ea3079e384-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:54114
I20260620 14:30:58.865219 10444 tablet_copy_service.cc:433] P 8bd903a427b648c59251747d73ab05d3: ending tablet copy session 3c362b2ae7f64faaa35124ea3079e384-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c362b2ae7f64faaa35124ea3079e384
I20260620 14:30:58.872681 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:58.874424 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.874543 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.875067 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.876904 10888 server_base.cc:1034] running on GCE node
I20260620 14:30:58.877087 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:58.877408 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.878532 10888 hybrid_clock.cc:648] HybridClock initialized: now 1781965858878514 us; error 43 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.880127 10888 init.cc:377] Logged in from keytab as oryx/127.4.58.125@KRBTEST.COM (short username oryx)
I20260620 14:30:58.880610 10888 webserver.cc:460] Webserver started at http://127.4.58.125:33705/ using document root <none> and password file <none>
I20260620 14:30:58.880869 10888 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.880913 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.882249 10888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:58.883221 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:58.883491 10888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:58.883607 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "3c362b2ae7f64faaa35124ea3079e384"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "cdd04f12290cfc9f2895994704c5dcb4"
server_key_iv: "c7d60d28815b7d7c0af4d96aac003037"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.883746 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:58.899154 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:58.901476 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:58.906042 10888 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.125:34289
I20260620 14:30:58.906080 11059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.125:34289 every 8 connection(s)
I20260620 14:30:58.906776 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:58.908587 11060 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:30:58.909687 11060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Bootstrap starting.
I20260620 14:30:58.920595 11060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:30:58.920953 11060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Bootstrap complete.
I20260620 14:30:58.921525 11060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } }
I20260620 14:30:58.921658 11060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c362b2ae7f64faaa35124ea3079e384, State: Initialized, Role: LEARNER
I20260620 14:30:58.921751 11060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } }
I20260620 14:30:58.922117 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } }
I20260620 14:30:58.922221 11060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:58.922315 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:30:58.925683 11074 catalog_manager.cc:1259] Loaded cluster ID: fe63a31684494869a3b4facc8e0746b9
I20260620 14:30:58.925750 11074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: loading cluster ID for follower catalog manager: success
I20260620 14:30:58.927851 11074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: acquiring CA information for follower catalog manager: success
I20260620 14:30:58.929279 11074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:30:58.967160 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:58.967428 10988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:58.967474 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:58.971482 10988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:58.971552 10988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:58.971570 10988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:58.971585 10988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:58.971601 10988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:58.977077 10988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.124.keytab
--principal=oryx/127.4.58.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.58.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:58.978402 10988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:58.979384 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:58.988013 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.988054 10988 server_base.cc:1034] running on GCE node
W20260620 14:30:58.988026 11079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:58.988255 11080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:58.988900 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:58.990015 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:58.991186 10988 hybrid_clock.cc:648] HybridClock initialized: now 1781965858991155 us; error 44 us; skew 500 ppm
Jun 20 14:30:58 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:58.993971 10988 init.cc:377] Logged in from keytab as oryx/127.4.58.124@KRBTEST.COM (short username oryx)
I20260620 14:30:58.994957 10988 webserver.cc:460] Webserver started at http://127.4.58.124:38019/ using document root <none> and password file <none>
I20260620 14:30:58.995677 10988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:58.995749 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:58.995981 10988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:58.998162 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/instance:
uuid: "6e7fcb985e524467bc706682577f72dd"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "c83b4f33d231adde6319a6dcec8bd664"
server_key_iv: "715368fb6e0f212a07c5bd08647fb9af"
server_key_version: "encryptionkey@0"
I20260620 14:30:58.998706 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal/instance:
uuid: "6e7fcb985e524467bc706682577f72dd"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "c83b4f33d231adde6319a6dcec8bd664"
server_key_iv: "715368fb6e0f212a07c5bd08647fb9af"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.002714 10988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:30:59.005398 11089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.006820 10988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:59.006968 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "6e7fcb985e524467bc706682577f72dd"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "c83b4f33d231adde6319a6dcec8bd664"
server_key_iv: "715368fb6e0f212a07c5bd08647fb9af"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.007087 10988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.040385 10988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:59.043931 10988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:59.044126 10988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:59.051054 10988 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.124:45427
I20260620 14:30:59.051090 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.124:45427 every 8 connection(s)
I20260620 14:30:59.052031 10988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:59.054044 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 10988
I20260620 14:30:59.054858 11142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:59.059206 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.124@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:30:59.068610 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.055505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.4.58.124:45427)
0620 14:30:59.055917 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.055932 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.055970 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:59.056618 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.056636 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.057033 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.057333 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.058257 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.059315 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.060430 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.060951 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.061106 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.063942 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.063949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.063960 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.063988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.065754 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.066214 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.066219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.066224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.066334 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.066619 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.066622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.066623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.067041 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.067217 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.067491 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.067770 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"spinlock_wait_cycles":9472,"thread_start_us":122,"threads_started":1}
I20260620 14:30:59.070200 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.061282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.4.58.126:46443)
0620 14:30:59.061408 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.061413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.061697 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:30:59.061823 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.061827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.061877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.061964 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.062607 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.063406 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.064450 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.064669 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.064787 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.066906 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.066909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.066911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.066936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.068362 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.069014 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.069017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.069019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.069110 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.069370 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.069374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.069376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.069582 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.069673 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.069990 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.070091 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:30:59.070283 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.061120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54130)
0620 14:30:59.061559 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.061605 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.061726 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.062010 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.062016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.062360 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.062490 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.062496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.063562 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.063566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.064322 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.064331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.064791 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.064795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.064907 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.065329 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.065340 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.066766 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.068460 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.068468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.068474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.068677 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.069213 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.069215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.069217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.069293 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.069677 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.069683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.069798 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.069936 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.070107 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":158,"threads_started":1}
I20260620 14:30:59.071539 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:59.073093 10615 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.124@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:59.079602 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.071991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36196)
0620 14:30:59.072104 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.072117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.072197 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.072591 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.072593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.072721 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.072843 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.072847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.073962 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.073965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.074604 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.074613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.075051 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.075053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.075121 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.075548 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.075559 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.076718 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.078240 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.078243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.078244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.078412 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.078733 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.078735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.078737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.078809 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.079203 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.079205 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.079279 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.079348 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.079479 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:30:59.079674 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.072120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.4.58.125:34289)
0620 14:30:59.072362 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.072365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.072377 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:59.072413 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.072453 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.072494 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.072567 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.072963 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.073856 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.074721 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.074962 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.075066 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.076865 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.076868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.076870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.076895 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.078148 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.078553 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.078556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.078558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.078646 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.078915 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.078919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.078920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.079107 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.079215 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.079394 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.079547 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260620 14:30:59.080309 11142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.124@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
I20260620 14:30:59.087772 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.080893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.4.58.124:45427)
0620 14:30:59.081014 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.081018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.081029 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:59.081054 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.081057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.081088 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.081146 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.081445 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.082272 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.082970 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.083212 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.083295 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.085131 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.085135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.085138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.085159 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.086406 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.086752 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.086755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.086757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.086823 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.087037 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.087040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.087042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.087202 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.087280 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.087492 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.087602 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:59.087795 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.080722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36054)
0620 14:30:59.080862 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.080873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.080942 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.081165 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.081167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.081298 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.081355 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.081359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.082380 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.082383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.082869 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.082875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.083316 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.083319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.083388 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.083787 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.083800 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.084998 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.086505 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.086508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.086510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.086675 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.086893 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.086896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.086898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.086966 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.087280 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.087282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.087359 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.087475 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.087673 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":77}
I20260620 14:30:59.090723 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Bootstrap starting.
I20260620 14:30:59.093055 11142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:59.093782 11142 log.cc:824] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:59.095922 11142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: No bootstrap required, opened a new log
I20260620 14:30:59.098958 11142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } }
I20260620 14:30:59.099188 11142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:59.099223 11142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7fcb985e524467bc706682577f72dd, State: Initialized, Role: FOLLOWER
I20260620 14:30:59.099689 11142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } }
I20260620 14:30:59.100191 11148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } } }
I20260620 14:30:59.100325 11148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:59.100929 11142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:30:59.105690 11159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:59.105790 11159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:59.109025 10988 master_runner.cc:428] Detected that this master 6e7fcb985e524467bc706682577f72dd is joining an existing cluster
I20260620 14:30:59.109099 10988 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.58.124:45427
I20260620 14:30:59.119877 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.112528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.4.58.124:45427)
0620 14:30:59.112616 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.112619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.112629 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:59.112659 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.112661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.112694 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.112790 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.113395 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.114142 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115479 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115768 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115823 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.117190 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.117194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.117196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.117218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.118438 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.118808 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.118811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.118813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.118879 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.119110 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.119113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.119115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.119278 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.119359 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.119590 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.119733 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":39}
I20260620 14:30:59.119889 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.111501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36064)
0620 14:30:59.112085 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.112097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.112165 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.112879 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.112882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.113165 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.113281 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.113289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.114257 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.114260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.115389 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.115396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115926 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.116299 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.116692 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.116703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.117070 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.118545 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.118548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.118551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.118729 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.118961 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.118964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.118966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.119037 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.119359 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.119362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.119442 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.119536 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.119683 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":29,"thread_start_us":86,"threads_started":1}
I20260620 14:30:59.122056 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10486
I20260620 14:30:59.129402 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:36975
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:46443
--webserver_port=44113
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:59.149639 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.111915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36204)
0620 14:30:59.112527 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.112540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.112668 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.113043 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.113046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.113225 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.113310 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.113315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.114296 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.114298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.115231 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.115240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115915 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.116022 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.145397 (+ 29375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.145418 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.145967 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.147946 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.147949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.147952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.148191 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.148491 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.148496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.148576 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.149077 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.149080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.149177 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.149268 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.149441 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":72,"threads_started":1}
I20260620 14:30:59.149807 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.111961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.4.58.126:46443)
0620 14:30:59.112107 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.112112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.112126 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:30:59.112172 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.112175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.112226 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.112311 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.112908 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.113739 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115045 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115402 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115481 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.145990 (+ 30509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.145995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.145998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.146032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.148021 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148517 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.148521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.148523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.148603 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148914 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.148917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.148919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.149136 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.149202 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.149529 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.149648 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:30:59.150365 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.112196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.4.58.125:34289)
0620 14:30:59.112307 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.112311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.112608 (+ 297us) server_negotiation.cc:407] Connection header received
0620 14:30:59.112813 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.112816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.112864 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.112912 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.113423 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.114184 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115354 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115703 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115769 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.146177 (+ 30408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.146182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.146184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.146225 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.147803 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148311 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.148314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.148316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.148397 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148666 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.148669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.148671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.148944 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.149035 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.150086 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.150214 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:59.150502 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.111461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54138)
0620 14:30:59.111778 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.111792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.111894 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.112324 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.112348 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.112578 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.112734 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.112743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.113897 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.113900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.114887 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.114897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.115655 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.115659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.115767 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.116271 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.116285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.145745 (+ 29460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.148166 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.148171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.148408 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.148707 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.148709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.148711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.148816 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.149268 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.149271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.149360 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.150183 (+ 823us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.150330 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":29,"thread_start_us":125,"threads_started":1}
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.126@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
I20260620 14:30:59.164484 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.155002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36080)
0620 14:30:59.155170 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.155185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.155276 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.155684 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.155688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.155913 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.156029 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.156036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.157145 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.157148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.158059 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.158071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.158790 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.158794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.158897 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.159410 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.159424 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.161072 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.162889 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.162893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.162896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.163152 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.163441 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.163444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.163446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.163540 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.163930 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.163933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.164045 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.164145 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.164304 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 14:30:59.164502 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.155111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.4.58.124:45427)
0620 14:30:59.155433 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.155437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.155448 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:59.155511 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.155514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.155566 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.155640 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.156144 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.157012 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.158172 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.158528 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.158643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.161221 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.161225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.161228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.161257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.162773 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.163250 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.163253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.163255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.163350 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.163624 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.163627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.163629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.163846 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.163939 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.164219 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.164347 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260620 14:30:59.165894 10424 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.58.124:45427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6e7fcb985e524467bc706682577f72dd"
member_type: NON_VOTER
last_known_addr {
host: "127.4.58.124"
port: 45427
}
attrs {
promote: true
}
}
dest_uuid: "8bd903a427b648c59251747d73ab05d3"
cas_config_opid_index: 5
I20260620 14:30:59.166530 10424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } }
I20260620 14:30:59.168023 10984 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6e7fcb985e524467bc706682577f72dd (127.4.58.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.169539 10984 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.58.124:45427
I20260620 14:30:59.169651 11116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:59.169719 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.169821 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.170405 10988 master.cc:537] Master@127.4.58.124:45427 shutting down...
I20260620 14:30:59.171052 11116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:30:59.171888 10983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:30:59.174604 10410 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 -> Peer 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427): Couldn't send request to peer 6e7fcb985e524467bc706682577f72dd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:30:59.175065 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.167740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36212)
0620 14:30:59.167871 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.167886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.167974 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.171150 (+ 3176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.171153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.171173 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.171261 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.171267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.172318 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.172321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.173554 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.173566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.174649 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.174653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.174744 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.174790 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.174859 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.174937 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:30:59.175460 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.168901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.4.58.125:34289)
0620 14:30:59.170744 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.170748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.170765 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:30:59.170821 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.170824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.170884 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.170980 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.171402 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.172168 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.173696 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.174327 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.174433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.174586 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.175257 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.175349 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1744}
I20260620 14:30:59.176224 11034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 LEARNER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:30:59.177644 10983 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:30:59.179392 10988 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:30:59.179668 10988 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:30:59.179764 10988 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: stopping tablet replica
I20260620 14:30:59.179618 11034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6e7fcb985e524467bc706682577f72dd (127.4.58.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.181573 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd903a427b648c59251747d73ab05d3. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.181679 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:30:59.181746 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: NON_VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: true } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.181799 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:30:59.184166 10983 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: attempting to promote NON_VOTER 3c362b2ae7f64faaa35124ea3079e384 to VOTER
I20260620 14:30:59.184746 10988 master.cc:552] Master@127.4.58.124:45427 shutdown complete.
I20260620 14:30:59.184813 10988 master_runner.cc:305] Clearing existing system tablet
I20260620 14:30:59.184711 10983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } }
I20260620 14:30:59.185040 10988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:59.185084 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:59.186298 10988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:59.186699 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.186197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36084)
0620 14:30:59.186526 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.186614 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:45427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250}
I20260620 14:30:59.187088 11173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.187249 10988 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.187343 10988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:59.187372 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "6e7fcb985e524467bc706682577f72dd"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "c83b4f33d231adde6319a6dcec8bd664"
server_key_iv: "715368fb6e0f212a07c5bd08647fb9af"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.187707 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.187212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36088)
0620 14:30:59.187593 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.187644 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:45427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314}
W20260620 14:30:59.187850 10410 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 -> Peer 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427): Couldn't send request to peer 6e7fcb985e524467bc706682577f72dd. Status: Network error: Client connection negotiation failed: client connection to 127.4.58.124:45427: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:30:59.188310 11034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 LEARNER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:30:59.188740 10984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:30:59.189399 10988 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:30:59.190991 10984 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.191799 10988 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:30:59.191877 10988 log.cc:1196] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:30:59.192111 10988 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Deleting consensus metadata
I20260620 14:30:59.192248 10988 master_runner.cc:315] Copying system tablet from 127.4.58.126:46443
I20260620 14:30:59.192464 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.192564 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.192668 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.192725 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.193408 10988 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: tablet copy: Beginning tablet copy session from remote peer at address 127.4.58.126:46443
I20260620 14:30:59.193516 11034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.194977 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.195102 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:59.202888 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.194321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54152)
0620 14:30:59.194607 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.194622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.194754 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.195130 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.195132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.195352 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.195466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.195474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.196589 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.196592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.197536 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.197547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.198248 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.198251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.198348 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.198886 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.198901 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.199333 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.201241 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.201244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.201247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.201516 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.201790 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.201794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.201796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.201979 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.202356 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.202359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.202465 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.202562 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.202707 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":61,"threads_started":1}
I20260620 14:30:59.202929 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.194125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.4.58.126:46443)
0620 14:30:59.194234 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.194239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.194685 (+ 446us) server_negotiation.cc:407] Connection header received
0620 14:30:59.194849 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.194853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.194898 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.194970 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.195596 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.196436 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.197673 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.198018 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.198088 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.199485 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.199489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.199492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.199517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.201139 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.201615 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.201619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.201621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.201704 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.202053 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.202057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.202059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.202269 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.202358 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.202644 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.202769 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:30:59.203799 10444 tablet_copy_service.cc:140] P 8bd903a427b648c59251747d73ab05d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6e7fcb985e524467bc706682577f72dd ({username='oryx', principal='oryx/127.4.58.124@KRBTEST.COM'} at 127.0.0.1:54152)
I20260620 14:30:59.203904 10444 tablet_copy_service.cc:161] P 8bd903a427b648c59251747d73ab05d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6e7fcb985e524467bc706682577f72dd at {username='oryx', principal='oryx/127.4.58.124@KRBTEST.COM'} at 127.0.0.1:54152: session id = 6e7fcb985e524467bc706682577f72dd-00000000000000000000000000000000
I20260620 14:30:59.205179 10444 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:30:59.206434 10988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:30:59.208819 10988 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: tablet copy: Starting download of 0 data blocks...
I20260620 14:30:59.209017 10988 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: tablet copy: Starting download of 1 WAL segments...
I20260620 14:30:59.210433 10988 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:30:59.212203 10444 tablet_copy_service.cc:342] P 8bd903a427b648c59251747d73ab05d3: Request end of tablet copy session 6e7fcb985e524467bc706682577f72dd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.58.124@KRBTEST.COM'} at 127.0.0.1:54152
I20260620 14:30:59.212289 10444 tablet_copy_service.cc:433] P 8bd903a427b648c59251747d73ab05d3: ending tablet copy session 6e7fcb985e524467bc706682577f72dd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6e7fcb985e524467bc706682577f72dd
I20260620 14:30:59.219794 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:59.221745 11180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.222008 11183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.221850 11181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.222618 10988 server_base.cc:1034] running on GCE node
I20260620 14:30:59.222826 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:59.223148 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:59.224303 10988 hybrid_clock.cc:648] HybridClock initialized: now 1781965859224288 us; error 33 us; skew 500 ppm
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:59.225860 10988 init.cc:377] Logged in from keytab as oryx/127.4.58.124@KRBTEST.COM (short username oryx)
I20260620 14:30:59.226348 10988 webserver.cc:460] Webserver started at http://127.4.58.124:40731/ using document root <none> and password file <none>
I20260620 14:30:59.226526 10988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:59.226567 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:59.228016 10988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:59.229111 11190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.229410 10988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:30:59.229485 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "6e7fcb985e524467bc706682577f72dd"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "c83b4f33d231adde6319a6dcec8bd664"
server_key_iv: "715368fb6e0f212a07c5bd08647fb9af"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.229570 10988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.247004 10988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:59.249275 10988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:59.254101 10988 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.124:45427
I20260620 14:30:59.254125 11242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.124:45427 every 8 connection(s)
I20260620 14:30:59.254670 10988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:59.256361 11243 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:30:59.257598 11243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Bootstrap starting.
I20260620 14:30:59.268540 11243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:30:59.268859 11243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: Bootstrap complete.
I20260620 14:30:59.269419 11243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } }
I20260620 14:30:59.269556 11243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6e7fcb985e524467bc706682577f72dd, State: Initialized, Role: LEARNER
I20260620 14:30:59.269652 11243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } }
I20260620 14:30:59.269979 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.270090 11243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:59.270184 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: LEARNER
I20260620 14:30:59.273103 11260 catalog_manager.cc:1259] Loaded cluster ID: fe63a31684494869a3b4facc8e0746b9
I20260620 14:30:59.273232 11260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: loading cluster ID for follower catalog manager: success
W20260620 14:30:59.275058 11168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:59.275393 11168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260620 14:30:59.275377 11260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: acquiring CA information for follower catalog manager: success
W20260620 14:30:59.275475 11168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:30:59.276906 11260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:30:59.280251 11168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:59.280303 11168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:59.280364 11168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:30:59.283161 10749 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
I20260620 14:30:59.285960 11168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:36975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=44113
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:59.287047 11168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:59.288065 11168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:59.297399 11262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.297394 11263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.297508 11168 server_base.cc:1034] running on GCE node
W20260620 14:30:59.297399 11265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.298079 11168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:59.298696 11168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:59.299856 11168 hybrid_clock.cc:648] HybridClock initialized: now 1781965859299831 us; error 35 us; skew 500 ppm
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:59.302611 11168 init.cc:377] Logged in from keytab as oryx/127.4.58.65@KRBTEST.COM (short username oryx)
I20260620 14:30:59.303457 11168 webserver.cc:460] Webserver started at http://127.4.58.65:44113/ using document root <none> and password file <none>
I20260620 14:30:59.303999 11168 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:59.304080 11168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:59.307955 11168 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:59.310211 11272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.311470 11168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:59.311590 11168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "b57256c51e4e73b4291844b19da347ad"
server_key_iv: "d8b56b4c6025b2600aecaebc08805c8b"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.312160 11168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.325996 11168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:59.329107 11168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:59.329288 11168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:59.329959 11168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:59.330880 11168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:59.330928 11168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.330981 11168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:59.331012 11168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.339745 11168 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:36975
I20260620 14:30:59.339771 11385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:36975 every 8 connection(s)
I20260620 14:30:59.340822 11168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:30:59.350272 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 11168
I20260620 14:30:59.350471 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10620
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:59.357219 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.342585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:34601 (local address 127.4.58.126:46443)
0620 14:30:59.342750 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.342755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.343779 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 14:30:59.344926 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.344929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.344978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.345026 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.346476 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.347287 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348450 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348807 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348895 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.352618 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.352622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.352625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.352654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.354612 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355329 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.355421 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355827 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.356066 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.356135 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.356895 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.357060 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:30:59.357219 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.344242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:47601 (local address 127.4.58.125:34289)
0620 14:30:59.344362 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.344366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.344478 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:30:59.344889 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.344893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.344948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.344999 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.346464 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.346987 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348475 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348690 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348746 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.353262 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.353266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.353275 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.353308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.354754 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355329 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.355421 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355847 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.356070 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.356139 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.356912 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.357054 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:30:59.358357 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.343137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:34601)
0620 14:30:59.343592 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.343645 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.344641 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.345204 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.345213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.345737 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.346309 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.346318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.347401 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.347404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.348342 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.348352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348956 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348968 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.349287 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.350003 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.350015 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.352409 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.354732 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.354735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.354737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355218 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.355528 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.355533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355727 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.356206 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.356209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.356583 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.357463 (+ 880us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.357602 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":236,"spinlock_wait_cycles":114560,"thread_start_us":129,"threads_started":1}
I20260620 14:30:59.358764 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.342681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:56303 (local address 127.4.58.124:45427)
0620 14:30:59.342945 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.342949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.343742 (+ 793us) server_negotiation.cc:407] Connection header received
0620 14:30:59.344854 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.344858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.344914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.345003 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.346439 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.347505 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348386 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348725 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348839 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.352618 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.352622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.352625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.352654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.354545 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355328 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.355425 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355847 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.355850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.355852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.356070 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.356150 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.358464 (+ 2314us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.358614 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260620 14:30:59.358355 11391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.343137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.4.58.65:56303)
0620 14:30:59.343592 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.343646 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.344642 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.345205 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.345214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.345732 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.346299 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.346312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.347601 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.347604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.348272 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.348281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348997 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.349001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.349165 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.349819 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.349841 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.352409 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.354705 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.354712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.354731 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355217 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.355530 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.355534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355740 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.356202 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.356208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.356582 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.356854 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.357214 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":106,"spinlock_wait_cycles":29824,"thread_start_us":105,"threads_started":1}
I20260620 14:30:59.359361 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.344149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.4.58.65:47601)
0620 14:30:59.344391 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.344404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.344657 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.345204 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.345213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.345734 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.346311 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.346318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.347126 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.347128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.348345 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.348352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.348956 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.348959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.349185 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.349959 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.349975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.353057 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.354864 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.354868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.354870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355217 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.355528 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.355531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.355533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.355740 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.356202 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.356208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.356582 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.357792 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.357941 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":237,"spinlock_wait_cycles":51968,"thread_start_us":83,"threads_started":1}
I20260620 14:30:59.359910 11387 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:59.360155 11387 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.360522 11386 heartbeater.cc:347] Connected to a master server at 127.4.58.124:45427
I20260620 14:30:59.360534 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:40625
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:46443
--webserver_port=38177
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:59.360600 11386 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.360697 11387 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
I20260620 14:30:59.360791 11386 heartbeater.cc:511] Master 127.4.58.124:45427 requested a full tablet report, sending...
I20260620 14:30:59.361980 11388 heartbeater.cc:347] Connected to a master server at 127.4.58.125:34289
I20260620 14:30:59.361972 10424 ts_manager.cc:194] Re-registered known tserver with Master: 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975)
I20260620 14:30:59.362068 11388 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.362218 11388 heartbeater.cc:511] Master 127.4.58.125:34289 requested a full tablet report, sending...
I20260620 14:30:59.362231 11207 ts_manager.cc:194] Registered new tserver with Master: 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975)
I20260620 14:30:59.362694 10424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:34601
I20260620 14:30:59.363217 11024 ts_manager.cc:194] Registered new tserver with Master: 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975)
W20260620 14:30:59.472957 11394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:59.473250 11394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:59.473295 11394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:59.477427 11394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:59.477489 11394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:59.477552 11394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:30:59.482887 11394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:40625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=38177
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:59.484028 11394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:59.485082 11394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:59.495039 11399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.495038 11402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.495030 11400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.495244 11394 server_base.cc:1034] running on GCE node
I20260620 14:30:59.495647 11394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:59.496373 11394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:59.497543 11394 hybrid_clock.cc:648] HybridClock initialized: now 1781965859497525 us; error 31 us; skew 500 ppm
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:59.500922 11394 init.cc:377] Logged in from keytab as oryx/127.4.58.66@KRBTEST.COM (short username oryx)
I20260620 14:30:59.501852 11394 webserver.cc:460] Webserver started at http://127.4.58.66:38177/ using document root <none> and password file <none>
I20260620 14:30:59.502383 11394 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:59.502454 11394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:59.506537 11394 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:30:59.508704 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.509986 11394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:59.510084 11394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "d75c336580736bd352051b0cc673cc7a"
server_key_iv: "ad8c2a020acca03dda6af63e9735f6a9"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.510638 11394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.518043 10883 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
I20260620 14:30:59.533437 11394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:59.536494 11394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:59.536652 11394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:59.537300 11394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:59.538153 11394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:59.538200 11394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.538239 11394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:59.538251 11394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.547176 11394 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:40625
I20260620 14:30:59.547264 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:40625 every 8 connection(s)
I20260620 14:30:59.548247 11394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:30:59.557974 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 11394
I20260620 14:30:59.558182 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10754
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:59.565456 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.550071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:60933 (local address 127.4.58.125:34289)
0620 14:30:59.550232 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.550237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.551162 (+ 925us) server_negotiation.cc:407] Connection header received
0620 14:30:59.552310 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.552313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.552368 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.552441 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.553792 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554512 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556315 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556650 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556729 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.561418 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.561422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.561425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.561453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.562873 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563560 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.563564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.563566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.563652 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.564044 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.564047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.564049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.564270 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.564335 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.565169 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.565291 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:30:59.565471 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.551636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:58417 (local address 127.4.58.126:46443)
0620 14:30:59.551758 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.551762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.551930 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:30:59.552320 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.552323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.552447 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.552501 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.553791 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554617 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556315 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556662 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556737 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.561660 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.561675 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.561678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.561714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.563542 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.564006 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.564010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.564011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.564095 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.564372 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.564376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.564377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.564584 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.564651 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.565173 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.565312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:30:59.565896 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:34777
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:46443
--webserver_port=39127
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:59.566599 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.550527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.4.58.66:34411)
0620 14:30:59.551007 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.551062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.552120 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.552640 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.552643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.553113 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.553645 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.553656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554733 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.554736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.556176 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.556187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556955 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.557335 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.558315 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.558328 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.560904 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.563163 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.563169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.563462 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.563784 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.563789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.563944 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.564413 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.564420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.564857 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.565105 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.565471 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":442,"spinlock_wait_cycles":39168,"thread_start_us":102,"threads_started":1}
I20260620 14:30:59.566596 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.550527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.4.58.66:60933)
0620 14:30:59.551007 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.551062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.552112 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.552628 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.552635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.553073 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.553631 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.553646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554689 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.554692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.556170 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.556181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556955 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.557327 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.558137 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.558160 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.561259 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.563014 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.563041 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.563448 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.563750 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.563754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.563936 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.564410 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.564416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.564855 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.565604 (+ 749us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.565743 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":297,"spinlock_wait_cycles":32384,"thread_start_us":126,"threads_started":1}
I20260620 14:30:59.567461 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.551546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34411 (local address 127.4.58.124:45427)
0620 14:30:59.551658 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.551663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.551678 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:59.552406 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.552409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.552461 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.552524 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.553794 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554601 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556315 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556671 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556786 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.561136 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.561141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.561143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.561177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.563036 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563590 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.563594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.563596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.563686 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.564036 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.564039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.564041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.564269 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.564367 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.567146 (+ 2779us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.567285 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:30:59.567915 11525 heartbeater.cc:347] Connected to a master server at 127.4.58.125:34289
I20260620 14:30:59.568183 11525 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.568742 11525 heartbeater.cc:511] Master 127.4.58.125:34289 requested a full tablet report, sending...
I20260620 14:30:59.569566 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.551541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:58417)
0620 14:30:59.551819 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.551838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.552119 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.552640 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.552643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.553109 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.553659 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.553666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.554736 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.554738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.556176 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.556187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.556955 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.556959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.557335 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.558651 (+ 1316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.558665 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.561490 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.563660 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.563663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.563665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.563890 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.564203 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.564206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.564208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.564280 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.564716 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.564719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.564857 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.569162 (+ 4305us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.569324 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":301,"spinlock_wait_cycles":9856,"thread_start_us":74,"threads_started":1}
I20260620 14:30:59.569937 11024 ts_manager.cc:194] Registered new tserver with Master: c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:30:59.570207 11523 heartbeater.cc:347] Connected to a master server at 127.4.58.124:45427
I20260620 14:30:59.570271 11524 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:59.570299 11523 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.570317 11524 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.570501 11524 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
I20260620 14:30:59.570498 11523 heartbeater.cc:511] Master 127.4.58.124:45427 requested a full tablet report, sending...
I20260620 14:30:59.571110 10424 ts_manager.cc:194] Re-registered known tserver with Master: c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:30:59.571802 10424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:58417
I20260620 14:30:59.572274 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.566635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36100)
0620 14:30:59.566894 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.566908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.566982 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.567786 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.567789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.567810 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.567901 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.567907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.568911 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.568914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.570033 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.570042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.571834 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.571838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.571942 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.571977 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.572060 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.572151 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 14:30:59.572355 11207 ts_manager.cc:194] Registered new tserver with Master: c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:30:59.572634 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.567129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.4.58.124:45427)
0620 14:30:59.567518 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.567522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.567534 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:59.567574 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.567576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.567630 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.567762 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.568044 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.568793 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.570999 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.571698 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.571804 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.571961 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.572419 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.572508 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":91,"threads_started":1}
I20260620 14:30:59.573318 11217 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:30:59.573822 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd903a427b648c59251747d73ab05d3. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.573935 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: LEARNER
W20260620 14:30:59.691126 11530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:59.691373 11530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:59.691416 11530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:59.695691 11530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:59.695755 11530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:59.695816 11530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:30:59.701287 11530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:34777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=39127
--tserver_master_addrs=127.4.58.126:46443,127.4.58.125:34289,127.4.58.124:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:59.702296 11530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:59.703306 11530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:59.712657 11537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:59.712620 11536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.712697 11530 server_base.cc:1034] running on GCE node
W20260620 14:30:59.712633 11539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:59.713406 11530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:59.714057 11530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:59.715224 11530 hybrid_clock.cc:648] HybridClock initialized: now 1781965859715200 us; error 34 us; skew 500 ppm
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:59.718114 11530 init.cc:377] Logged in from keytab as oryx/127.4.58.67@KRBTEST.COM (short username oryx)
I20260620 14:30:59.719013 11530 webserver.cc:460] Webserver started at http://127.4.58.67:39127/ using document root <none> and password file <none>
I20260620 14:30:59.719558 11530 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:59.719638 11530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:59.723734 11530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:30:59.725764 11546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.727013 11530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:59.727128 11530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
format_stamp: "Formatted at 2026-06-20 14:30:58 on dist-test-slave-gqzk"
server_key: "4e201563e402938f868e1b0d8ea9ccaf"
server_key_iv: "f2de99f356141baf30421b88ac30cba0"
server_key_version: "encryptionkey@0"
I20260620 14:30:59.727670 11530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:59.745724 11530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:59.748925 11530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:59.749104 11530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:59.749687 11530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:30:59.750543 11530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:30:59.750591 11530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.750643 11530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:30:59.750667 11530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:59.759521 11530 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34777
I20260620 14:30:59.759549 11659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34777 every 8 connection(s)
I20260620 14:30:59.760519 11530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:30:59.763947 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 11530
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.125@KRBTEST.COM
I20260620 14:30:59.776273 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.765779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.4.58.126:46443)
0620 14:30:59.766254 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.766257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.766272 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:59.766324 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.766327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.766371 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.767348 (+ 977us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.767448 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.768149 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.769365 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.769698 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.771484 (+ 1786us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.771604 (+ 120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.771608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.771610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.771628 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.773513 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.774368 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.774374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.774376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.774473 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.774793 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.774796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.774798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.775033 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.775854 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.775932 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.776066 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":84,"threads_started":1}
I20260620 14:30:59.778869 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.762317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:47839 (local address 127.4.58.126:46443)
0620 14:30:59.762516 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.762521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.763387 (+ 866us) server_negotiation.cc:407] Connection header received
0620 14:30:59.764556 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.764560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.764613 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.764663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.766930 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.767673 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.769522 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.769838 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.769924 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.773763 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.773767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.773769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.773796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.775474 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.776109 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.776113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.776115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.776208 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.776580 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.776583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.776585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.776805 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.776883 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.778435 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.778713 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 14:30:59 dist-test-slave-gqzk krb5kdc[10379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.124@KRBTEST.COM
I20260620 14:30:59.780519 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.762763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:47839)
0620 14:30:59.763235 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.763289 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.764319 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.765000 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.765009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.765713 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.766456 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.766477 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.767834 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.767838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.769413 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.769421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.770142 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.770147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.770422 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.771060 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.771108 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.773572 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.775591 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.775594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.775603 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.776005 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.776292 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.776295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.776297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.776485 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.776932 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.776939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.777301 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.777574 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.779048 (+ 1474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":150,"spinlock_wait_cycles":31488,"thread_start_us":130,"threads_started":1}
I20260620 14:30:59.781826 11661 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46443
I20260620 14:30:59.782055 11661 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.782570 11661 heartbeater.cc:511] Master 127.4.58.126:46443 requested a full tablet report, sending...
I20260620 14:30:59.780047 10424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:30:59.784154 10424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:30:59.784288 10423 ts_manager.cc:194] Re-registered known tserver with Master: d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777)
I20260620 14:30:59.784493 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.763887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:38853 (local address 127.4.58.125:34289)
0620 14:30:59.763990 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.763993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.764238 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:30:59.764541 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.764544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.764599 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.764650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.767115 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.767897 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.769596 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.769918 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.769996 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.773861 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.773865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.773867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.773895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.775442 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.776732 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.776735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.776737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.776863 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.777181 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.777185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.777186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.778230 (+ 1044us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.778580 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.784073 (+ 5493us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.784198 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":812,"server-negotiator.queue_time_us":46}
I20260620 14:30:59.785144 10423 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:47839
I20260620 14:30:59.785527 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.763757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.4.58.67:38853)
0620 14:30:59.764145 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.764158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.764320 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.765000 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.765009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.765967 (+ 958us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.766999 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.767006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.768020 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.768023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.769424 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.769434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.770142 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.770145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.770423 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.771942 (+ 1519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.771957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.773698 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.775572 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.775580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.775601 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.776623 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.776971 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.776974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.776975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.777080 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.778624 (+ 1544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.778627 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.783815 (+ 5188us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.785112 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.785341 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1163,"spinlock_wait_cycles":3328,"thread_start_us":72,"threads_started":1}
I20260620 14:30:59.786285 11662 heartbeater.cc:347] Connected to a master server at 127.4.58.125:34289
I20260620 14:30:59.786384 11662 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.787710 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.766720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.4.58.125:34289)
0620 14:30:59.772587 (+ 5867us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.772592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.772605 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:59.772644 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.772647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.772698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.774744 (+ 2046us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.779030 (+ 4286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.779757 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.781910 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.782262 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.782363 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.783932 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.783936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.783939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.783964 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.785745 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.786208 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.786211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.786213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.786301 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.786582 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.786585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.786586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.786775 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.786880 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.787429 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.787555 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":846,"server-negotiator.queue_time_us":5778,"thread_start_us":87,"threads_started":1}
I20260620 14:30:59.787773 11662 heartbeater.cc:511] Master 127.4.58.125:34289 requested a full tablet report, sending...
I20260620 14:30:59.787907 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: NON_VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: true } } }
I20260620 14:30:59.788030 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.788678 11024 ts_manager.cc:194] Registered new tserver with Master: d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777)
I20260620 14:30:59.790091 11664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.762763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.4.58.67:41951)
0620 14:30:59.763235 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.763289 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.764323 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.765000 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.765009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.767112 (+ 2103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:59.767216 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.767222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.768187 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.768190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.769424 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.769434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.777672 (+ 8238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.777676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.777780 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.778213 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:59.778227 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:59.779742 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:59.781731 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.781734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.781736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.788531 (+ 6795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.788944 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:59.788947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:59.788949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:59.789113 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:59.789595 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:59.789598 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:59.789682 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.789758 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.789910 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":101,"spinlock_wait_cycles":31488,"thread_start_us":120,"threads_started":1}
I20260620 14:30:59.790567 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.762357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:41951 (local address 127.4.58.124:45427)
0620 14:30:59.762518 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.762523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.763387 (+ 864us) server_negotiation.cc:407] Connection header received
0620 14:30:59.764617 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.764621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.764673 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.764733 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.767333 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.768078 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.777082 (+ 9004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.777414 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.777520 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.779894 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.779898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.779901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.779941 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.781619 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.788641 (+ 7022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.788645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.788647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.788745 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.789233 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.789237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.789239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.789467 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.789543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.790319 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.790461 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:30:59.791404 11660 heartbeater.cc:347] Connected to a master server at 127.4.58.124:45427
I20260620 14:30:59.791487 11660 heartbeater.cc:464] Registering TS with master...
I20260620 14:30:59.791636 11660 heartbeater.cc:511] Master 127.4.58.124:45427 requested a full tablet report, sending...
I20260620 14:30:59.792248 11207 ts_manager.cc:194] Registered new tserver with Master: d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777)
I20260620 14:30:59.793035 11171 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: attempting to promote NON_VOTER 6e7fcb985e524467bc706682577f72dd to VOTER
I20260620 14:30:59.793478 11171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:30:59.793817 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.766520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.4.58.124:45427)
0620 14:30:59.766622 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.766625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.773028 (+ 6403us) server_negotiation.cc:407] Connection header received
0620 14:30:59.774201 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.774205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.774262 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.774315 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:59.778151 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.778850 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.782496 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.782820 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.782895 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.790432 (+ 7537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:59.790436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:59.790438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:59.790463 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:59.792124 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.792562 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.792565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.792567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.792646 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:59.792920 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:59.792924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:59.792925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:59.793111 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:59.793225 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.793563 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.793680 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:59.794840 11217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 LEARNER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:30:59.794829 11034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Refusing update from remote peer 8bd903a427b648c59251747d73ab05d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:30:59.795177 11682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:30:59.795296 11171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:30:59.796965 11171 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6e7fcb985e524467bc706682577f72dd (127.4.58.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.797442 11034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6e7fcb985e524467bc706682577f72dd (127.4.58.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.798440 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.798538 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.798650 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.798820 11685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.798945 11685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:59.798547 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.799376 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.799468 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:59.799726 11217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6e7fcb985e524467bc706682577f72dd (127.4.58.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.801455 11249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd903a427b648c59251747d73ab05d3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:30:59.801560 11249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:59.810312 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.805363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34777 (local address 127.0.0.1:51560)
0620 14:30:59.805496 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.805512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.805600 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.806438 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.806441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.806465 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.806567 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.806573 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.807875 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.807878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.808603 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.808611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.809891 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.809895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.810022 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.810049 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.810093 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.810159 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":7}
I20260620 14:30:59.810948 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.805532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.4.58.67:34777)
0620 14:30:59.805911 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.805924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.805950 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:30:59.806035 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.806042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.806298 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.806489 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.806653 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.807784 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.808717 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.809757 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.810511 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.810665 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.810718 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.810808 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260620 14:30:59.811157 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.806282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:40625 (local address 127.0.0.1:56914)
0620 14:30:59.807032 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.807044 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.807134 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.807523 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.807525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.807538 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.807602 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.807606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.808839 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.808843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.809963 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.809973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.810779 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.810783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.810866 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.810888 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.810958 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.811027 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":54,"thread_start_us":77,"threads_started":1}
I20260620 14:30:59.811197 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.806658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.4.58.66:40625)
0620 14:30:59.806963 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.806974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.807125 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:30:59.807234 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.807239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.807404 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.807552 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.807680 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.808724 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.810071 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.810678 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.810768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.810908 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.810991 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.811076 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260620 14:30:59.812727 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.805991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36975 (local address 127.0.0.1:49338)
0620 14:30:59.806221 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.806234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.806315 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.807255 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.807257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.807269 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.807356 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.807363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.808789 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.808792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.810007 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.810015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.812373 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.812377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.812464 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.812488 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.812534 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.812599 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":52,"thread_start_us":103,"threads_started":1}
I20260620 14:30:59.813421 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.806283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.4.58.65:36975)
0620 14:30:59.806825 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.806838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.806861 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:30:59.806939 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.806944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.807145 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.807288 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.807441 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.808631 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.810132 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.812255 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.813078 (+ 823us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.813204 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.813247 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.813321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":103,"threads_started":1}
I20260620 14:30:59.814172 11594 tablet_service.cc:1467] Processing CreateTablet for tablet 64a0e4d0da184960a6d656965f2321c1 (DEFAULT_TABLE table=test-table [id=0e2d2062650242aca8b921314c4cf3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:59.814167 11457 tablet_service.cc:1467] Processing CreateTablet for tablet 64a0e4d0da184960a6d656965f2321c1 (DEFAULT_TABLE table=test-table [id=0e2d2062650242aca8b921314c4cf3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:59.815367 11594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a0e4d0da184960a6d656965f2321c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:59.815394 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a0e4d0da184960a6d656965f2321c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:59.819521 11320 tablet_service.cc:1467] Processing CreateTablet for tablet 64a0e4d0da184960a6d656965f2321c1 (DEFAULT_TABLE table=test-table [id=0e2d2062650242aca8b921314c4cf3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:30:59.820453 11320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a0e4d0da184960a6d656965f2321c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:59.824148 11692 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Bootstrap starting.
I20260620 14:30:59.825057 11693 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Bootstrap starting.
I20260620 14:30:59.826849 11693 tablet_bootstrap.cc:654] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:59.826881 11692 tablet_bootstrap.cc:654] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:59.827678 11693 log.cc:824] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:59.828186 11692 log.cc:824] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:59.828333 11695 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Bootstrap starting.
I20260620 14:30:59.830284 11692 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: No bootstrap required, opened a new log
I20260620 14:30:59.830371 11693 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: No bootstrap required, opened a new log
I20260620 14:30:59.830552 11692 ts_tablet_manager.cc:1397] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:30:59.830569 11693 ts_tablet_manager.cc:1397] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:30:59.830574 11695 tablet_bootstrap.cc:654] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:59.831351 11695 log.cc:824] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:59.833582 11695 tablet_bootstrap.cc:492] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: No bootstrap required, opened a new log
I20260620 14:30:59.834822 11695 ts_tablet_manager.cc:1397] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:30:59.834699 11692 raft_consensus.cc:348] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.835000 11692 raft_consensus.cc:374] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:59.834846 11693 raft_consensus.cc:348] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.835064 11692 raft_consensus.cc:729] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02d074ed9184c0d8aaa723ad74e1d13, State: Initialized, Role: FOLLOWER
I20260620 14:30:59.835115 11693 raft_consensus.cc:374] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:59.835163 11693 raft_consensus.cc:729] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5573f8a36db4294a5a16c16c61cc0f9, State: Initialized, Role: FOLLOWER
I20260620 14:30:59.835523 11692 consensus_queue.cc:260] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.835582 11693 consensus_queue.cc:260] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.836537 11692 ts_tablet_manager.cc:1428] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:30:59.836544 11693 ts_tablet_manager.cc:1428] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:30:59.836709 11661 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
I20260620 14:30:59.837481 11524 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
I20260620 14:30:59.844077 11695 raft_consensus.cc:348] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.844336 11695 raft_consensus.cc:374] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:59.844379 11695 raft_consensus.cc:729] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c73c64a8ec54a4e8396a2f5dbf2ea45, State: Initialized, Role: FOLLOWER
I20260620 14:30:59.844799 11695 consensus_queue.cc:260] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.845494 11387 heartbeater.cc:503] Master 127.4.58.126:46443 was elected leader, sending a full tablet report...
I20260620 14:30:59.845849 11695 ts_tablet_manager.cc:1428] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:30:59.910094 11700 raft_consensus.cc:482] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:30:59.910279 11700 raft_consensus.cc:504] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.911412 11700 leader_election.cc:288] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777), c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:30:59.915827 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.911669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34777 (local address 127.4.58.65:51193)
0620 14:30:59.912068 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.912081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.912186 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.912495 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.912498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.912515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.912613 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.912620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.913565 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.913568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.914677 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.914684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.915400 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.915404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.915517 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.915537 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.915599 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.915673 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":57,"thread_start_us":72,"threads_started":1}
I20260620 14:30:59.915827 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.911833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:51193 (local address 127.4.58.67:34777)
0620 14:30:59.911948 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.911953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.912157 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:30:59.912336 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.912339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.912391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.912463 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.912714 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.913455 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.914816 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.915288 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.915376 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.915442 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.915637 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.915705 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:59.916018 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.911789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:45979 (local address 127.4.58.66:40625)
0620 14:30:59.911975 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.911980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.912079 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:30:59.912251 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.912255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.912301 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.912371 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:30:59.912653 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.913279 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.914739 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.915363 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.915468 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.915530 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.915820 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.915893 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:30:59.916350 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.911668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:40625 (local address 127.4.58.65:45979)
0620 14:30:59.911997 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:59.912010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:59.912122 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:59.912413 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:59.912416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:59.912434 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:30:59.912538 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.912545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.913380 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.913383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.914628 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:59.914639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.915587 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.915590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:59.915701 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.915722 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:30:59.916138 (+ 416us) client_negotiation.cc:240] Negotiation successful
0620 14:30:59.916214 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":71,"threads_started":1}
I20260620 14:30:59.916980 11477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a0e4d0da184960a6d656965f2321c1" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" is_pre_election: true
I20260620 14:30:59.917280 11477 raft_consensus.cc:2454] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 0.
I20260620 14:30:59.917743 11275 leader_election.cc:302] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c73c64a8ec54a4e8396a2f5dbf2ea45, c5573f8a36db4294a5a16c16c61cc0f9; no voters:
I20260620 14:30:59.917768 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a0e4d0da184960a6d656965f2321c1" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" is_pre_election: true
I20260620 14:30:59.918026 11614 raft_consensus.cc:2454] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 0.
I20260620 14:30:59.918051 11700 raft_consensus.cc:2781] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:30:59.918149 11700 raft_consensus.cc:482] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:30:59.918201 11700 raft_consensus.cc:3037] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:59.919183 11700 raft_consensus.cc:504] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.919538 11700 leader_election.cc:288] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 election: Requested vote from peers d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777), c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:30:59.919875 11477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a0e4d0da184960a6d656965f2321c1" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
I20260620 14:30:59.919874 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a0e4d0da184960a6d656965f2321c1" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
I20260620 14:30:59.920007 11614 raft_consensus.cc:3037] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:59.920007 11477 raft_consensus.cc:3037] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:59.921298 11477 raft_consensus.cc:2454] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 1.
I20260620 14:30:59.921298 11614 raft_consensus.cc:2454] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 1.
I20260620 14:30:59.921591 11275 leader_election.cc:302] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3c73c64a8ec54a4e8396a2f5dbf2ea45, c5573f8a36db4294a5a16c16c61cc0f9, d02d074ed9184c0d8aaa723ad74e1d13; no voters:
I20260620 14:30:59.921808 11700 raft_consensus.cc:2781] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:59.921983 11700 raft_consensus.cc:686] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 LEADER]: Becoming Leader. State: Replica: 3c73c64a8ec54a4e8396a2f5dbf2ea45, State: Running, Role: LEADER
I20260620 14:30:59.922359 11700 consensus_queue.cc:237] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } }
I20260620 14:30:59.926304 10424 catalog_manager.cc:5495] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } health_report { overall_health: UNKNOWN } } }
I20260620 14:30:59.964560 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:59.961193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.4.58.65:36975)
0620 14:30:59.961307 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:59.961310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:59.961323 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:59.961501 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:59.961503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:59.961557 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:59.961645 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:59.961837 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.962339 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.963164 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:59.963371 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:59.963455 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:59.963750 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:59.964173 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:59.964299 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:59.964373 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:30:59.964424 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:30:59.971939 11614 raft_consensus.cc:1261] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Refusing update from remote peer 3c73c64a8ec54a4e8396a2f5dbf2ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:59.971939 11477 raft_consensus.cc:1261] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Refusing update from remote peer 3c73c64a8ec54a4e8396a2f5dbf2ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:59.972781 11703 consensus_queue.cc:1035] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:59.972977 11700 consensus_queue.cc:1035] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:59.982807 11708 mvcc.cc:205] Tried to move back new op lower bound from 7298932162435792896 to 7298932162245058560. Current Snapshot: MvccSnapshot[applied={T|T < 7298932162435792896}]
I20260620 14:30:59.993134 10424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:59.993325 10424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:30:59.997844 10424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bd903a427b648c59251747d73ab05d3: Sending DeleteTablet for 3 replicas of tablet 64a0e4d0da184960a6d656965f2321c1
I20260620 14:30:59.998840 11457 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a0e4d0da184960a6d656965f2321c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:59 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:56914
I20260620 14:30:59.999096 11594 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a0e4d0da184960a6d656965f2321c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:59 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51560
I20260620 14:30:59.999440 11320 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a0e4d0da184960a6d656965f2321c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:30:59 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:49338
I20260620 14:30:59.999738 11718 tablet_replica.cc:330] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: stopping tablet replica
I20260620 14:31:00.000151 11718 raft_consensus.cc:2229] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:00.000502 11718 raft_consensus.cc:2258] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:00.002080 11718 ts_tablet_manager.cc:1905] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:00.002765 11717 tablet_replica.cc:330] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: stopping tablet replica
I20260620 14:31:00.003219 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10392
I20260620 14:31:00.003381 11717 raft_consensus.cc:2229] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:00.004312 11717 raft_consensus.cc:2258] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:00.005196 11719 tablet_replica.cc:330] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: stopping tablet replica
I20260620 14:31:00.005774 11719 raft_consensus.cc:2229] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:00.006299 11719 raft_consensus.cc:2258] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:00.006835 11717 ts_tablet_manager.cc:1905] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:00.006906 11718 ts_tablet_manager.cc:1918] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:00.007015 11718 log.cc:1196] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/64a0e4d0da184960a6d656965f2321c1
I20260620 14:31:00.007326 11718 ts_tablet_manager.cc:1939] T 64a0e4d0da184960a6d656965f2321c1 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting consensus metadata
I20260620 14:31:00.010444 11717 ts_tablet_manager.cc:1918] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:00.010586 11717 log.cc:1196] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/64a0e4d0da184960a6d656965f2321c1
I20260620 14:31:00.010905 11717 ts_tablet_manager.cc:1939] T 64a0e4d0da184960a6d656965f2321c1 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting consensus metadata
W20260620 14:31:00.011366 11550 connection.cc:535] server connection from 127.0.0.1:51560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:31:00.014384 11719 ts_tablet_manager.cc:1905] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:31:00.014642 11413 connection.cc:441] server connection from 127.0.0.1:56914 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56914 (request call id 1) could send its response
I20260620 14:31:00.017722 11719 ts_tablet_manager.cc:1918] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:00.017967 11719 log.cc:1196] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/64a0e4d0da184960a6d656965f2321c1
I20260620 14:31:00.018383 11719 ts_tablet_manager.cc:1939] T 64a0e4d0da184960a6d656965f2321c1 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Deleting consensus metadata
W20260620 14:31:00.019695 11273 connection.cc:441] server connection from 127.0.0.1:49338 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49338 (request call id 1) could send its response
I20260620 14:31:00.024055 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.001309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.4.58.125:34289)
0620 14:31:00.001432 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:00.001437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:00.013224 (+ 11787us) server_negotiation.cc:407] Connection header received
0620 14:31:00.013489 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:00.013493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:00.013555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:00.013609 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:00.014663 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.015419 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.016846 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.017222 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.017342 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:00.019589 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:00.019593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:00.019596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:00.019628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:00.021873 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.022351 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.022355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.022357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.022559 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.022916 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.022919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.022921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.023172 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:00.023322 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:00.023720 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:31:00.023872 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:00.025803 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.015598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.4.58.124:45427)
0620 14:31:00.015721 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:00.015725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:00.015872 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:31:00.016060 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:00.016064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:00.016115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:00.016167 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:00.017999 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.019164 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.020289 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.020665 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.020799 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:00.022062 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:00.022066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:00.022069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:00.022101 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:00.023744 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.024149 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.024153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.024155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.024296 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.024561 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.024564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.024566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.024784 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:00.024906 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:00.025371 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:31:00.025500 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:31:00.035006 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.015623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.4.58.125:34289)
0620 14:31:00.015738 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:00.015743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:00.026112 (+ 10369us) server_negotiation.cc:407] Connection header received
0620 14:31:00.026297 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:00.026301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:00.026348 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:00.026388 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:00.027167 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.027678 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.028514 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.028739 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.028886 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:00.030977 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:00.030982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:00.030985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:00.031029 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:00.032986 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.033565 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.033569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.033571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.033666 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.034043 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.034047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.034049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.034309 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:00.034585 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:00.034680 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:31:00.034817 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:31:00.035501 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.001573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.4.58.124:45427)
0620 14:31:00.001678 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:00.001682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:00.024951 (+ 23269us) server_negotiation.cc:407] Connection header received
0620 14:31:00.025126 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:00.025129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:00.025178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:00.025238 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:00.026371 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.027161 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.028321 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:00.029069 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:00.029193 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:00.031384 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:00.031388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:00.031391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:00.031421 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:00.033634 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.034140 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.034144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.034146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.034231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:00.034540 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:00.034543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:00.034545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:00.034766 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:00.034863 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:00.035215 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:31:00.035343 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:31:00.987524 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.987000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:57135)
0620 14:31:00.987252 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:00.987378 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20260620 14:31:00.987896 11661 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:00.988818 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.987688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:38681)
0620 14:31:00.988373 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:00.988469 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":68,"threads_started":1}
W20260620 14:31:00.989181 11387 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:00.989568 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:00.989042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:46223)
0620 14:31:00.989399 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:00.989471 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
W20260620 14:31:00.989861 11524 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:46443 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.499044 11730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bd903a427b648c59251747d73ab05d3)
I20260620 14:31:01.499267 11730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:01.499773 11731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bd903a427b648c59251747d73ab05d3)
I20260620 14:31:01.500061 11731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:01.500555 11730 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125:34289)
I20260620 14:31:01.501072 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.500668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54594)
0620 14:31:01.500892 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.500974 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260620 14:31:01.501096 11731 leader_election.cc:288] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427)
W20260620 14:31:01.501233 11191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.501847 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.501352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54606)
0620 14:31:01.501686 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.501760 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":57,"threads_started":1}
W20260620 14:31:01.502091 11008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.502326 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.502093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54618)
0620 14:31:01.502215 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.502264 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:31:01.502419 11191 leader_election.cc:334] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.503175 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.502925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54622)
0620 14:31:01.503036 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.503110 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:31:01.503291 11008 leader_election.cc:334] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.506045 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.501564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.4.58.124:45427)
0620 14:31:01.501770 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:01.501785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:01.501796 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:01.501984 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:01.501987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:01.502033 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:01.502103 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:01.502573 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.503394 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.504853 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.505484 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.505590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:01.505630 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:01.505813 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:31:01.505909 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260620 14:31:01.506075 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.501087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.4.58.125:34289)
0620 14:31:01.501639 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:01.501643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:01.501657 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:01.501705 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:01.501707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:01.501753 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:01.502112 (+ 359us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:01.502210 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.502938 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.504678 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.505337 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.505455 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:01.505524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:01.505802 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:31:01.505967 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":475,"thread_start_us":56,"threads_started":1}
I20260620 14:31:01.506044 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.501002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:34289 (local address 127.0.0.1:36260)
0620 14:31:01.501187 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.501200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:01.501290 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:01.501873 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:01.501876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:01.501895 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:01.502011 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:01.502018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.503064 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.503067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:01.504555 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:01.504567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.505560 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.505565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:01.505659 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:01.505685 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:31:01.505783 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:31:01.505897 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":25,"thread_start_us":60,"threads_started":1}
I20260620 14:31:01.506090 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.501380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:45427 (local address 127.0.0.1:36146)
0620 14:31:01.501715 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.501727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:01.501812 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:01.502238 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:01.502241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:01.502257 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:01.502474 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:01.502484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.503490 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.503493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:01.504693 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:01.504702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:01.505616 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:01.505620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:01.505702 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:01.505724 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:31:01.505797 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:31:01.505915 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":34,"thread_start_us":46,"threads_started":1}
I20260620 14:31:01.506557 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e7fcb985e524467bc706682577f72dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c362b2ae7f64faaa35124ea3079e384" is_pre_election: true
I20260620 14:31:01.506780 11034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7fcb985e524467bc706682577f72dd in term 1.
I20260620 14:31:01.506853 11217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c362b2ae7f64faaa35124ea3079e384" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e7fcb985e524467bc706682577f72dd" is_pre_election: true
I20260620 14:31:01.507046 11217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c362b2ae7f64faaa35124ea3079e384 in term 1.
I20260620 14:31:01.507128 11191 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c362b2ae7f64faaa35124ea3079e384, 6e7fcb985e524467bc706682577f72dd; no voters: 8bd903a427b648c59251747d73ab05d3
I20260620 14:31:01.507341 11730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:31:01.507395 11730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8bd903a427b648c59251747d73ab05d3)
I20260620 14:31:01.507380 11010 leader_election.cc:302] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c362b2ae7f64faaa35124ea3079e384, 6e7fcb985e524467bc706682577f72dd; no voters: 8bd903a427b648c59251747d73ab05d3
I20260620 14:31:01.507447 11730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:01.507573 11731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:31:01.507629 11731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8bd903a427b648c59251747d73ab05d3)
I20260620 14:31:01.507668 11731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:01.508450 11730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:01.508589 11731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:01.508833 11730 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 election: Requested vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 3c362b2ae7f64faaa35124ea3079e384 (127.4.58.125:34289)
I20260620 14:31:01.508972 11731 leader_election.cc:288] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 election: Requested vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427)
I20260620 14:31:01.509189 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.508958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54634)
0620 14:31:01.509063 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.509118 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:31:01.509249 11217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c362b2ae7f64faaa35124ea3079e384" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e7fcb985e524467bc706682577f72dd"
I20260620 14:31:01.509435 11217 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c362b2ae7f64faaa35124ea3079e384 in current term 2: Already voted for candidate 6e7fcb985e524467bc706682577f72dd in this term.
I20260620 14:31:01.509465 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e7fcb985e524467bc706682577f72dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c362b2ae7f64faaa35124ea3079e384"
I20260620 14:31:01.509619 11034 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7fcb985e524467bc706682577f72dd in current term 2: Already voted for candidate 3c362b2ae7f64faaa35124ea3079e384 in this term.
I20260620 14:31:01.509846 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.509240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54650)
0620 14:31:01.509724 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.509781 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
I20260620 14:31:01.509963 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.509767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54666)
0620 14:31:01.509848 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.509894 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 14:31:01.510082 11191 leader_election.cc:334] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.510280 11191 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7fcb985e524467bc706682577f72dd; no voters: 3c362b2ae7f64faaa35124ea3079e384, 8bd903a427b648c59251747d73ab05d3
I20260620 14:31:01.510438 11730 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:31:01.510571 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.510269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54680)
0620 14:31:01.510396 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.510465 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:31:01.510661 11008 leader_election.cc:334] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:01.510727 11008 leader_election.cc:302] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c362b2ae7f64faaa35124ea3079e384; no voters: 6e7fcb985e524467bc706682577f72dd, 8bd903a427b648c59251747d73ab05d3
I20260620 14:31:01.510839 11731 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:31:01.989017 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.988523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:43887)
0620 14:31:01.988838 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.988905 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260620 14:31:01.990613 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.989715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:37187)
0620 14:31:01.990429 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.990511 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":62,"threads_started":1}
I20260620 14:31:01.990850 11742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:01.990405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:42935)
0620 14:31:01.990710 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:01.990770 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260620 14:31:02.990274 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:02.989836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:51269)
0620 14:31:02.990113 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:02.990178 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260620 14:31:02.991796 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:02.991316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:53431)
0620 14:31:02.991631 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:02.991695 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":71,"threads_started":1}
I20260620 14:31:02.992064 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:02.991578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:58115)
0620 14:31:02.991898 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:02.991969 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":66,"threads_started":1}
I20260620 14:31:03.449133 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:03.449280 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:03.449782 11746 leader_election.cc:288] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427)
I20260620 14:31:03.450309 11217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c362b2ae7f64faaa35124ea3079e384" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e7fcb985e524467bc706682577f72dd" is_pre_election: true
I20260620 14:31:03.450404 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.450022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54956)
0620 14:31:03.450248 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.450310 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260620 14:31:03.450452 11217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c362b2ae7f64faaa35124ea3079e384 in term 2.
I20260620 14:31:03.450762 11010 leader_election.cc:302] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c362b2ae7f64faaa35124ea3079e384, 6e7fcb985e524467bc706682577f72dd; no voters:
I20260620 14:31:03.450929 11746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:31:03.450999 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:03.451038 11746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:31:03.451297 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.451033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54962)
0620 14:31:03.451187 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.451235 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 14:31:03.451418 11008 leader_election.cc:334] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:03.452106 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:03.452396 11746 leader_election.cc:288] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 election: Requested vote from peers 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443), 6e7fcb985e524467bc706682577f72dd (127.4.58.124:45427)
I20260620 14:31:03.452775 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.452526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54974)
0620 14:31:03.452632 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.452682 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:31:03.452836 11217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c362b2ae7f64faaa35124ea3079e384" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6e7fcb985e524467bc706682577f72dd"
I20260620 14:31:03.452982 11217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:31:03.453374 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.453199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:54986)
0620 14:31:03.453286 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.453328 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:31:03.453527 11008 leader_election.cc:334] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
I20260620 14:31:03.454064 11217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c362b2ae7f64faaa35124ea3079e384 in term 3.
I20260620 14:31:03.454375 11010 leader_election.cc:302] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c362b2ae7f64faaa35124ea3079e384, 6e7fcb985e524467bc706682577f72dd; no voters: 8bd903a427b648c59251747d73ab05d3
I20260620 14:31:03.454543 11746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:31:03.454710 11746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [term 3 LEADER]: Becoming Leader. State: Replica: 3c362b2ae7f64faaa35124ea3079e384, State: Running, Role: LEADER
I20260620 14:31:03.454953 11746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } }
I20260620 14:31:03.456285 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c362b2ae7f64faaa35124ea3079e384. Latest consensus state: current_term: 3 leader_uuid: "3c362b2ae7f64faaa35124ea3079e384" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:31:03.456413 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:03.456682 11751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:03.458853 11751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:03.459210 11751 catalog_manager.cc:1259] Loaded cluster ID: fe63a31684494869a3b4facc8e0746b9
I20260620 14:31:03.459251 11751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:03.459751 11751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:03.460106 11751 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Loaded TSK: 0
I20260620 14:31:03.460742 11751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:03.516367 11023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:03.520102 11023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:03.523221 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.522954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:55006)
0620 14:31:03.523069 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.523132 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:31:03.523393 11217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [term 3 FOLLOWER]: Refusing update from remote peer 3c362b2ae7f64faaa35124ea3079e384: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:31:03.523895 11749 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:31:03.524041 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.523818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.0.0.1:55018)
0620 14:31:03.523920 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.523978 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:31:03.524133 11008 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 -> Peer 8bd903a427b648c59251747d73ab05d3 (127.4.58.126:46443): Couldn't send request to peer 8bd903a427b648c59251747d73ab05d3. Status: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:31:03.525391 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c362b2ae7f64faaa35124ea3079e384. Latest consensus state: current_term: 3 leader_uuid: "3c362b2ae7f64faaa35124ea3079e384" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:31:03.525516 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:03.526641 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3c362b2ae7f64faaa35124ea3079e384" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:31:03.526757 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:03.527354 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3c362b2ae7f64faaa35124ea3079e384" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8bd903a427b648c59251747d73ab05d3" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46443 } } peers { permanent_uuid: "3c362b2ae7f64faaa35124ea3079e384" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 34289 } attrs { promote: false } } peers { permanent_uuid: "6e7fcb985e524467bc706682577f72dd" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 45427 } attrs { promote: false } } }
I20260620 14:31:03.527618 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7fcb985e524467bc706682577f72dd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:31:03.536273 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.532490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34777 (local address 127.0.0.1:51574)
0620 14:31:03.532630 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.532645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:03.532731 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:03.533119 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:03.533122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:03.533141 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:03.533250 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.533258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.534270 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.534273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.535195 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.535205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.535893 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.535896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.535976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.535999 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:31:03.536065 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:31:03.536133 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":44}
I20260620 14:31:03.536639 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.532922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:40625 (local address 127.0.0.1:56918)
0620 14:31:03.533140 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.533154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:03.533234 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:03.533514 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:03.533517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:03.533530 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:03.533630 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.533635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.534547 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.534550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.535397 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.535412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.536242 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.536245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.536357 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.536385 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:31:03.536433 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:31:03.536499 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260620 14:31:03.537187 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.533025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.4.58.66:40625)
0620 14:31:03.533285 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.533290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.533303 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:03.533351 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.533353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.533398 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.533711 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:03.533811 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.534441 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.535512 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.536132 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.536860 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.536934 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.536975 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.537058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260620 14:31:03.537189 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.532581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.4.58.67:34777)
0620 14:31:03.532840 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.532844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.532859 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:31:03.532939 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.532942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.532995 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.533362 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:03.533474 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.534161 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.535324 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.535793 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.535883 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.535960 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.536988 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.537075 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260620 14:31:03.537794 11594 tablet_service.cc:1467] Processing CreateTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 (DEFAULT_TABLE table=test-table [id=6ed562f783924d24a386188bc0cdef22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:03.537798 11457 tablet_service.cc:1467] Processing CreateTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 (DEFAULT_TABLE table=test-table [id=6ed562f783924d24a386188bc0cdef22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:03.538075 11594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c4a06a4f8949a6a81b08fc5cab8a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:03.538089 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c4a06a4f8949a6a81b08fc5cab8a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:03.538444 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.533218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36975 (local address 127.0.0.1:49346)
0620 14:31:03.533422 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.533434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:03.533499 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:03.535116 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:03.535118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:03.535135 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:03.535199 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.535203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.536158 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.536161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.537173 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.537181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.538080 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.538083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.538165 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.538188 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:31:03.538252 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:31:03.538312 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260620 14:31:03.538837 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.533834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.4.58.65:36975)
0620 14:31:03.534056 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.534061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.534074 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:03.534121 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.534124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.534166 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.534247 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:03.535320 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.536030 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.537288 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.537855 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.537963 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.538022 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.538635 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.538716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260620 14:31:03.539391 11320 tablet_service.cc:1467] Processing CreateTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 (DEFAULT_TABLE table=test-table [id=6ed562f783924d24a386188bc0cdef22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:03.539654 11320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c4a06a4f8949a6a81b08fc5cab8a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:03.540688 11765 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Bootstrap starting.
I20260620 14:31:03.540690 11764 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Bootstrap starting.
I20260620 14:31:03.541591 11764 tablet_bootstrap.cc:654] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:03.541649 11765 tablet_bootstrap.cc:654] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:03.542111 11766 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Bootstrap starting.
I20260620 14:31:03.542870 11765 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: No bootstrap required, opened a new log
I20260620 14:31:03.542868 11764 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: No bootstrap required, opened a new log
I20260620 14:31:03.542951 11764 ts_tablet_manager.cc:1397] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:03.542953 11765 ts_tablet_manager.cc:1397] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:03.543005 11766 tablet_bootstrap.cc:654] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:03.543329 11765 raft_consensus.cc:348] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.543427 11765 raft_consensus.cc:374] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:03.543457 11765 raft_consensus.cc:729] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5573f8a36db4294a5a16c16c61cc0f9, State: Initialized, Role: FOLLOWER
I20260620 14:31:03.543388 11764 raft_consensus.cc:348] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.543509 11764 raft_consensus.cc:374] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:03.543537 11764 raft_consensus.cc:729] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02d074ed9184c0d8aaa723ad74e1d13, State: Initialized, Role: FOLLOWER
I20260620 14:31:03.543579 11765 consensus_queue.cc:260] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.543637 11764 consensus_queue.cc:260] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.543879 11765 ts_tablet_manager.cc:1428] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:03.545096 11764 ts_tablet_manager.cc:1428] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:03.546032 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.544205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:46831)
0620 14:31:03.545855 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.545920 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1585,"thread_start_us":102,"threads_started":1}
I20260620 14:31:03.546571 11525 heartbeater.cc:503] Master 127.4.58.125:34289 was elected leader, sending a full tablet report...
I20260620 14:31:03.546622 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.544238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:40433)
0620 14:31:03.546464 (+ 2226us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.546522 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2159,"thread_start_us":118,"threads_started":1}
I20260620 14:31:03.546868 11662 heartbeater.cc:503] Master 127.4.58.125:34289 was elected leader, sending a full tablet report...
I20260620 14:31:03.547174 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.546515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:43051)
0620 14:31:03.547054 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.547105 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":116,"threads_started":1}
I20260620 14:31:03.547609 11766 tablet_bootstrap.cc:492] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: No bootstrap required, opened a new log
I20260620 14:31:03.547708 11766 ts_tablet_manager.cc:1397] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 14:31:03.548132 11766 raft_consensus.cc:348] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.548241 11766 raft_consensus.cc:374] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:03.548269 11766 raft_consensus.cc:729] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c73c64a8ec54a4e8396a2f5dbf2ea45, State: Initialized, Role: FOLLOWER
I20260620 14:31:03.548382 11766 consensus_queue.cc:260] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.548673 11766 ts_tablet_manager.cc:1428] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:03.549467 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.548965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:45397)
0620 14:31:03.549322 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.549368 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":87,"threads_started":1}
I20260620 14:31:03.550042 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.549861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:40771)
0620 14:31:03.549953 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.549994 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:31:03.553090 11526 tablet.cc:2369] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:03.554419 11388 heartbeater.cc:503] Master 127.4.58.125:34289 was elected leader, sending a full tablet report...
W20260620 14:31:03.595752 11389 tablet.cc:2369] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:03.765810 11663 tablet.cc:2369] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:03.771297 11775 raft_consensus.cc:482] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:03.771443 11775 raft_consensus.cc:504] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.771857 11775 leader_election.cc:288] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777), c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:31:03.772356 11477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" is_pre_election: true
I20260620 14:31:03.772365 11771 raft_consensus.cc:482] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:03.772365 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" is_pre_election: true
I20260620 14:31:03.772552 11477 raft_consensus.cc:2454] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 0.
I20260620 14:31:03.772540 11771 raft_consensus.cc:504] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.772898 11275 leader_election.cc:302] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c73c64a8ec54a4e8396a2f5dbf2ea45, c5573f8a36db4294a5a16c16c61cc0f9; no voters:
I20260620 14:31:03.773005 11614 raft_consensus.cc:2454] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 0.
I20260620 14:31:03.773085 11775 raft_consensus.cc:2781] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:03.773157 11775 raft_consensus.cc:482] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:03.773205 11775 raft_consensus.cc:3037] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:03.773897 11771 leader_election.cc:288] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625), 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975)
I20260620 14:31:03.774573 11775 raft_consensus.cc:504] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.775095 11775 leader_election.cc:288] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 election: Requested vote from peers d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777), c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625)
I20260620 14:31:03.775414 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02d074ed9184c0d8aaa723ad74e1d13"
I20260620 14:31:03.775447 11477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5573f8a36db4294a5a16c16c61cc0f9"
I20260620 14:31:03.775537 11614 raft_consensus.cc:3037] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:03.775560 11477 raft_consensus.cc:3037] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:03.776721 11477 raft_consensus.cc:2454] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 1.
I20260620 14:31:03.776722 11614 raft_consensus.cc:2454] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in term 1.
I20260620 14:31:03.777137 11275 leader_election.cc:302] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c73c64a8ec54a4e8396a2f5dbf2ea45, c5573f8a36db4294a5a16c16c61cc0f9; no voters:
I20260620 14:31:03.777307 11775 raft_consensus.cc:2781] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:03.777513 11775 raft_consensus.cc:686] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [term 1 LEADER]: Becoming Leader. State: Replica: 3c73c64a8ec54a4e8396a2f5dbf2ea45, State: Running, Role: LEADER
I20260620 14:31:03.777673 11775 consensus_queue.cc:237] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } }
I20260620 14:31:03.778568 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.774339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36975 (local address 127.4.58.67:43503)
0620 14:31:03.774519 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.774533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:03.774621 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:03.774818 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:03.774821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:03.774836 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:03.775002 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.775009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.775938 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.775941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.777413 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.777423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.778175 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.778178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.778274 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.778299 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:31:03.778366 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:03.778438 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":139}
I20260620 14:31:03.778731 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.774090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:40625 (local address 127.4.58.67:33239)
0620 14:31:03.774209 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.774225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:03.774340 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:03.774801 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:03.774804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:03.774833 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:03.774976 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.774984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.775928 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.775931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.777420 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:03.777430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.778325 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.778329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:03.778433 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.778454 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:31:03.778521 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:03.778603 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":133}
I20260620 14:31:03.779263 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.774532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:43503 (local address 127.4.58.65:36975)
0620 14:31:03.774639 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.774643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.774656 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:03.774701 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.774703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.774745 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.774808 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:03.775126 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.775821 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.777549 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.778071 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.778169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.778233 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.779090 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.779161 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:03.779367 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.774163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:33239 (local address 127.4.58.66:40625)
0620 14:31:03.774501 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.774506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.774518 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:03.774564 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.774567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.774616 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.774687 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:03.775106 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.775795 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.777553 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.778091 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.778184 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.778246 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.779192 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.779269 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260620 14:31:03.779366 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.778685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:51187)
0620 14:31:03.779243 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.779296 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496}
I20260620 14:31:03.779855 11340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" is_pre_election: true
I20260620 14:31:03.780061 11477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c4a06a4f8949a6a81b08fc5cab8a79" candidate_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" is_pre_election: true
I20260620 14:31:03.780206 11477 raft_consensus.cc:2379] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d02d074ed9184c0d8aaa723ad74e1d13 in current term 1: Already voted for candidate 3c73c64a8ec54a4e8396a2f5dbf2ea45 in this term.
I20260620 14:31:03.780176 11023 catalog_manager.cc:5495] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c73c64a8ec54a4e8396a2f5dbf2ea45" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36975 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:03.780634 11549 leader_election.cc:302] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d02d074ed9184c0d8aaa723ad74e1d13; no voters: 3c73c64a8ec54a4e8396a2f5dbf2ea45, c5573f8a36db4294a5a16c16c61cc0f9
I20260620 14:31:03.780936 11771 raft_consensus.cc:2726] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:31:03.831503 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.828288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.4.58.65:36975)
0620 14:31:03.828411 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:03.828414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:03.828457 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:31:03.828532 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:03.828534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:03.828579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:03.828644 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:03.828950 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.829581 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.830465 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:03.830716 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:03.830808 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:03.831030 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:03.831097 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:03.831170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:03.831291 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:31:03.831343 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:03.833636 11477 raft_consensus.cc:1261] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Refusing update from remote peer 3c73c64a8ec54a4e8396a2f5dbf2ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:03.833712 11614 raft_consensus.cc:1261] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Refusing update from remote peer 3c73c64a8ec54a4e8396a2f5dbf2ea45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:03.834156 11775 consensus_queue.cc:1035] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d02d074ed9184c0d8aaa723ad74e1d13" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:03.834328 11777 consensus_queue.cc:1035] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5573f8a36db4294a5a16c16c61cc0f9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:03.835891 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.835602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:51939)
0620 14:31:03.835750 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.835807 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 14:31:03.836058 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.835827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:33519)
0620 14:31:03.835926 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.835981 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:31:03.839879 11783 mvcc.cc:205] Tried to move back new op lower bound from 7298932178258182144 to 7298932178034655232. Current Snapshot: MvccSnapshot[applied={T|T < 7298932178258182144}]
I20260620 14:31:03.840078 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.839692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:48947)
0620 14:31:03.839939 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.839998 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 14:31:03.841419 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.841144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.67:58253)
0620 14:31:03.841278 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.841340 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:31:03.842371 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.841970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.66:41967)
0620 14:31:03.842245 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.842298 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260620 14:31:03.843680 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.843467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46443 (local address 127.4.58.65:49699)
0620 14:31:03.843565 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.843619 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:31:03.845772 11023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:03.845935 11023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:03.849924 11023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c362b2ae7f64faaa35124ea3079e384: Sending DeleteTablet for 3 replicas of tablet 60c4a06a4f8949a6a81b08fc5cab8a79
I20260620 14:31:03.850720 11457 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:03 UTC) from {username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:56918
I20260620 14:31:03.850867 11594 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:03 UTC) from {username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:51574
I20260620 14:31:03.850857 11320 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c4a06a4f8949a6a81b08fc5cab8a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:03 UTC) from {username='oryx', principal='oryx/127.4.58.125@KRBTEST.COM'} at 127.0.0.1:49346
I20260620 14:31:03.851284 11789 tablet_replica.cc:330] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: stopping tablet replica
I20260620 14:31:03.851418 11789 raft_consensus.cc:2229] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:03.851490 11790 tablet_replica.cc:330] T 60c4a06a4f8949a6a81b08fc5cab8a79 P 3c73c64a8ec54a4e8396a2f5dbf2ea45: stopping tablet replica
I20260620 14:31:03.851576 11789 raft_consensus.cc:2258] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:03.851703 11791 tablet_replica.cc:330] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: stopping tablet replica
I20260620 14:31:03.851708 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 11168
I20260620 14:31:03.851820 11791 raft_consensus.cc:2229] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:03.851935 11791 raft_consensus.cc:2258] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:03.852854 11791 ts_tablet_manager.cc:1905] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:03.853166 11789 ts_tablet_manager.cc:1905] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:03.855481 11791 ts_tablet_manager.cc:1918] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:03.855482 11789 ts_tablet_manager.cc:1918] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:03.855566 11791 log.cc:1196] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/60c4a06a4f8949a6a81b08fc5cab8a79
I20260620 14:31:03.855566 11789 log.cc:1196] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/60c4a06a4f8949a6a81b08fc5cab8a79
I20260620 14:31:03.855818 11789 ts_tablet_manager.cc:1939] T 60c4a06a4f8949a6a81b08fc5cab8a79 P c5573f8a36db4294a5a16c16c61cc0f9: Deleting consensus metadata
I20260620 14:31:03.855815 11791 ts_tablet_manager.cc:1939] T 60c4a06a4f8949a6a81b08fc5cab8a79 P d02d074ed9184c0d8aaa723ad74e1d13: Deleting consensus metadata
I20260620 14:31:03.856525 11011 catalog_manager.cc:4841] TS d02d074ed9184c0d8aaa723ad74e1d13 (127.4.58.67:34777): tablet 60c4a06a4f8949a6a81b08fc5cab8a79 (table test-table [id=6ed562f783924d24a386188bc0cdef22]) successfully deleted
I20260620 14:31:03.856565 11010 catalog_manager.cc:4841] TS c5573f8a36db4294a5a16c16c61cc0f9 (127.4.58.66:40625): tablet 60c4a06a4f8949a6a81b08fc5cab8a79 (table test-table [id=6ed562f783924d24a386188bc0cdef22]) successfully deleted
I20260620 14:31:03.861753 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 11394
I20260620 14:31:03.862143 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:03.861752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36975 (local address 127.0.0.1:49366)
0620 14:31:03.861940 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:03.862023 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:36975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:31:03.862534 11010 catalog_manager.cc:4584] TS 3c73c64a8ec54a4e8396a2f5dbf2ea45 (127.4.58.65:36975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60c4a06a4f8949a6a81b08fc5cab8a79: Network error: Client connection negotiation failed: client connection to 127.4.58.65:36975: connect: Connection refused (error 111)
I20260620 14:31:03.866989 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 11530
I20260620 14:31:03.872037 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10888
I20260620 14:31:03.877236 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 10988
2026-06-20T14:31:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7371 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:03 dist-test-slave-gqzk krb5kdc[11794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:03 dist-test-slave-gqzk krb5kdc[11794](info): set up 2 sockets
Jun 20 14:31:03 dist-test-slave-gqzk krb5kdc[11794](info): commencing operation
krb5kdc: starting...
W20260620 14:31:04.837162 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:31:04 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab.
Entry for principal oryx/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab.
I20260620 14:31:04.955727 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39097
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab
--principal=oryx/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:05.066128 11807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:05.066421 11807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:05.066476 11807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:05.070437 11807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:05.070498 11807 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:31:05.070521 11807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:05.070544 11807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:05.070566 11807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:05.070593 11807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:05.075824 11807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39097
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.126.keytab
--principal=oryx/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:05.076894 11807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:05.077931 11807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:05.086262 11812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.086342 11813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.086263 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.086608 11807 server_base.cc:1034] running on GCE node
I20260620 14:31:05.087155 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:05.088217 11807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:05.089407 11807 hybrid_clock.cc:648] HybridClock initialized: now 1781965865089359 us; error 34 us; skew 500 ppm
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:05.092177 11807 init.cc:377] Logged in from keytab as oryx/127.4.58.126@KRBTEST.COM (short username oryx)
I20260620 14:31:05.093122 11807 webserver.cc:460] Webserver started at http://127.4.58.126:38351/ using document root <none> and password file <none>
I20260620 14:31:05.093691 11807 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:05.093755 11807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:05.093930 11807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:05.095927 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "b2f3f17cc05a4baa9d212a8f99804dea"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "fd7b8806b07b7c886fe704867474e800"
server_key_iv: "b708a34abf8bbac22b69d2855496b401"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.096426 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "b2f3f17cc05a4baa9d212a8f99804dea"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "fd7b8806b07b7c886fe704867474e800"
server_key_iv: "b708a34abf8bbac22b69d2855496b401"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.100039 11807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:31:05.102522 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.104022 11807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:31:05.104154 11807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "b2f3f17cc05a4baa9d212a8f99804dea"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "fd7b8806b07b7c886fe704867474e800"
server_key_iv: "b708a34abf8bbac22b69d2855496b401"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.104262 11807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:05.114349 11807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:05.117385 11807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:05.117542 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:05.125417 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:39097 every 8 connection(s)
I20260620 14:31:05.125445 11807 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:39097
I20260620 14:31:05.126547 11807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:05.129441 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:05.132346 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 11807
I20260620 14:31:05.132454 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:31:05.132741 4329 external_mini_cluster.cc:1428] Setting key d751a22c9a5156a245cd2eac5e5ec22a
I20260620 14:31:05.135583 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Bootstrap starting.
I20260620 14:31:05.138347 11885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:05.139173 11885 log.cc:824] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:05.141219 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: No bootstrap required, opened a new log
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:05.145321 11885 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } }
I20260620 14:31:05.145601 11885 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:05.145661 11885 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f3f17cc05a4baa9d212a8f99804dea, State: Initialized, Role: FOLLOWER
I20260620 14:31:05.146149 11885 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } }
I20260620 14:31:05.146312 11885 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:05.146414 11885 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:05.146557 11885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:05.147629 11885 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } }
I20260620 14:31:05.147670 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.134365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.4.58.126:39097)
0620 14:31:05.134785 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.134804 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.134848 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:31:05.135665 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.135688 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.136115 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.136499 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.137202 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.138678 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.139540 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.140077 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.140227 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.142588 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.142595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.142612 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.142644 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.144542 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.144991 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.144998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.145004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.145143 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.145430 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.145433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.145435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.145945 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.146140 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.146371 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.146698 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260620 14:31:05.148005 11885 leader_election.cc:302] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f3f17cc05a4baa9d212a8f99804dea; no voters:
I20260620 14:31:05.148305 11885 leader_election.cc:288] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:05.148401 11890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:05.148571 11890 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [term 1 LEADER]: Becoming Leader. State: Replica: b2f3f17cc05a4baa9d212a8f99804dea, State: Running, Role: LEADER
I20260620 14:31:05.149605 11885 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:05.149943 11890 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } }
I20260620 14:31:05.151487 11891 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } } }
I20260620 14:31:05.151970 11891 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:05.151664 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f3f17cc05a4baa9d212a8f99804dea. Latest consensus state: current_term: 1 leader_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f3f17cc05a4baa9d212a8f99804dea" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39097 } } }
I20260620 14:31:05.152336 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea [sys.catalog]: This master's current role is: LEADER
W20260620 14:31:05.154570 11904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:31:05.154635 11904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:31:05.154758 11905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:05.155635 11905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:05.162271 11905 catalog_manager.cc:1347] Generated new cluster ID: 9169656f050d472a925cfd6dcb078ec5
I20260620 14:31:05.162359 11905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:05.176347 11905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:05.177402 11905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:05.190397 11905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Generated new TSK 0
I20260620 14:31:05.191237 11905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
I20260620 14:31:05.220213 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:05.332479 11911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:05.332783 11911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:05.332846 11911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:05.336902 11911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:05.336962 11911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:05.337018 11911 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:05.337095 11911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:05.342602 11911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:05.343703 11911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:05.344794 11911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:05.354806 11911 server_base.cc:1034] running on GCE node
W20260620 14:31:05.354743 11916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.354730 11917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.354766 11919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.355418 11911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:05.356055 11911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:05.357230 11911 hybrid_clock.cc:648] HybridClock initialized: now 1781965865357180 us; error 66 us; skew 500 ppm
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:05.359966 11911 init.cc:377] Logged in from keytab as oryx/127.4.58.65@KRBTEST.COM (short username oryx)
I20260620 14:31:05.360901 11911 webserver.cc:460] Webserver started at http://127.4.58.65:33801/ using document root <none> and password file <none>
I20260620 14:31:05.361450 11911 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:05.361515 11911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:05.361701 11911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:05.363740 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "8ca54498f9b9d05f7b26c1c1162fa468"
server_key_iv: "29fda5a5df84a1e40c1675bfface757d"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.364256 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "8ca54498f9b9d05f7b26c1c1162fa468"
server_key_iv: "29fda5a5df84a1e40c1675bfface757d"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.368032 11911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:31:05.370510 11926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.371714 11911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:05.371829 11911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "8ca54498f9b9d05f7b26c1c1162fa468"
server_key_iv: "29fda5a5df84a1e40c1675bfface757d"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.371923 11911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:05.401213 11911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:05.404196 11911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:05.404369 11911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:05.405951 11911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:05.406000 11911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.406051 11911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:05.406080 11911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:05.416047 11911 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:43083
I20260620 14:31:05.416107 12041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:43083 every 8 connection(s)
I20260620 14:31:05.417212 11911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:05.419929 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.406846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:47295 (local address 127.4.58.126:39097)
0620 14:31:05.406966 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.406970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.408160 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 14:31:05.409356 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.409360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.409417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.409554 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.411030 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.411832 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.413350 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.413594 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.413691 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.416428 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.416432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.416435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.416464 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.418046 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.418582 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.418585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.418586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.418666 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.418933 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.418935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.418936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.419138 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.419210 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.419705 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.419808 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:31:05.420725 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.407429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.65:47295)
0620 14:31:05.408011 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.408062 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.409122 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.409755 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.409765 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.410199 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.410856 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.410878 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.411996 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.411999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.413226 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.413235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.413749 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.413752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.413931 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.414589 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.414608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.416248 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.418173 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.418178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.418189 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.418436 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.418739 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.418741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.418742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.418871 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.419237 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.419242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.419511 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.419673 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.419926 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
I20260620 14:31:05.421944 12042 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39097
I20260620 14:31:05.422168 12042 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:05.422704 12042 heartbeater.cc:511] Master 127.4.58.126:39097 requested a full tablet report, sending...
I20260620 14:31:05.424209 11839 ts_manager.cc:194] Registered new tserver with Master: b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083)
I20260620 14:31:05.425385 11839 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:47295
I20260620 14:31:05.426896 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 11911
I20260620 14:31:05.427032 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:05.427357 4329 external_mini_cluster.cc:1428] Setting key a68f6eb2d393fa75510cebeb3c058e42
I20260620 14:31:05.432180 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.424185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:51399 (local address 127.4.58.126:39097)
0620 14:31:05.425940 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.425944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.425957 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:05.425990 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.425992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.426025 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.426088 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.426567 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.427147 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.428033 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.428247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.428330 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.429404 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.429408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.429411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.429439 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.430757 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.431124 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.431128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.431130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.431210 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.431451 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.431454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.431456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.431620 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.431701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.431943 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.432042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1680}
I20260620 14:31:05.432197 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.424078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.65:51399)
0620 14:31:05.424353 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.424368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.424467 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.426126 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.426129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.426316 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.426459 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.426466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.427256 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.427260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.427933 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.427942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.428341 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.428345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.428445 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.428975 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.428987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.429281 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.430848 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.430852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.430854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.431040 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.431292 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.431295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.431297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.431372 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.431702 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.431706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.431791 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.431875 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.432014 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
I20260620 14:31:05.455925 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:05.565599 12051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:05.565883 12051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:05.565928 12051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:05.569926 12051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:05.569993 12051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:05.570039 12051 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:05.570073 12051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:05.575470 12051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:05.576620 12051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:05.577744 12051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:05.588219 12056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.588408 12051 server_base.cc:1034] running on GCE node
W20260620 14:31:05.588256 12059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.588266 12057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.589037 12051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:05.589730 12051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:05.590896 12051 hybrid_clock.cc:648] HybridClock initialized: now 1781965865590874 us; error 39 us; skew 500 ppm
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:05.593870 12051 init.cc:377] Logged in from keytab as oryx/127.4.58.66@KRBTEST.COM (short username oryx)
I20260620 14:31:05.594882 12051 webserver.cc:460] Webserver started at http://127.4.58.66:33973/ using document root <none> and password file <none>
I20260620 14:31:05.595468 12051 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:05.595539 12051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:05.595750 12051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:05.597983 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "76547a2fb1f54b1b97715b128b157cd0"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "239d88c7c82d9d83d8d73e9ad15ecfed"
server_key_iv: "0f804bd4cfb2bcc27904532bb0e47147"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.598567 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "76547a2fb1f54b1b97715b128b157cd0"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "239d88c7c82d9d83d8d73e9ad15ecfed"
server_key_iv: "0f804bd4cfb2bcc27904532bb0e47147"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.602536 12051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:05.605262 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.606616 12051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:05.606736 12051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "76547a2fb1f54b1b97715b128b157cd0"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "239d88c7c82d9d83d8d73e9ad15ecfed"
server_key_iv: "0f804bd4cfb2bcc27904532bb0e47147"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.606848 12051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:05.622066 12051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:05.625344 12051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:05.625537 12051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:05.627005 12051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:05.627063 12051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.627120 12051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:05.627144 12051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:05.637300 12051 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:34545
I20260620 14:31:05.637719 12181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:34545 every 8 connection(s)
I20260620 14:31:05.638325 12051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:05.640357 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.627899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:43389 (local address 127.4.58.126:39097)
0620 14:31:05.628017 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.628021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.629135 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 14:31:05.630191 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.630195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.630265 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.630339 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.631838 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.632616 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.633872 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.634096 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.634187 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.636839 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.636842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.636844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.636873 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.638311 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.638916 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.638920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.638922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.639031 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.639389 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.639392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.639394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.639589 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.639670 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.640133 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.640237 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:31:05.641193 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.628441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.66:43389)
0620 14:31:05.628980 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.629025 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.629970 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.630548 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.630557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.631015 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.631688 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.631709 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.632731 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.632734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.633747 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.633756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.634241 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.634245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.634417 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.635035 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.635056 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.636623 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.638427 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.638433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.638448 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.638796 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.639136 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.639139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.639141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.639294 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.639675 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.639680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.639907 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.640086 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.640337 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260620 14:31:05.642252 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12051
I20260620 14:31:05.642375 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:05.642653 4329 external_mini_cluster.cc:1428] Setting key 09b7a2ede207b7a9f2fd14b0fb74e5c7
I20260620 14:31:05.644352 12182 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39097
I20260620 14:31:05.644498 12182 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:05.645117 12182 heartbeater.cc:511] Master 127.4.58.126:39097 requested a full tablet report, sending...
I20260620 14:31:05.646111 11836 ts_manager.cc:194] Registered new tserver with Master: 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:05.646873 11836 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:43389
WARNING: no policy specified for oryx/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:31:05.652875 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.644618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:46009 (local address 127.4.58.126:39097)
0620 14:31:05.644729 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.644734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.645811 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 14:31:05.645906 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.645909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.645953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.646013 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.646465 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.647184 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.648044 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.648256 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.648342 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.649445 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.649449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.649452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.649476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.651134 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.651558 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.651562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.651564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.651645 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.651956 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.651959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.651961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.652229 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.652326 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.652579 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.652694 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:31:05.652890 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.644590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.66:46009)
0620 14:31:05.645718 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.645733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.645831 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.646047 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.646049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.646259 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.646361 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.646367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.647306 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.647309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.647928 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.647938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.648356 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.648359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.648451 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.648980 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.648991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.649313 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.651249 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.651253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.651256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.651463 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.651773 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.651776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.651778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.651870 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.652316 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.652319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.652417 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.652515 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.652670 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
I20260620 14:31:05.672832 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39097
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:05.782100 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:05.782382 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:05.782447 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:05.786495 12191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:05.786563 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:05.786618 12191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:31:05.786670 12191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:05.791906 12191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.58.126:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:05.792990 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:05.793993 12191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:05.803725 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.803807 12191 server_base.cc:1034] running on GCE node
W20260620 14:31:05.803726 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:05.803719 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:05.804327 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:05.805019 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:05.806185 12191 hybrid_clock.cc:648] HybridClock initialized: now 1781965865806163 us; error 31 us; skew 500 ppm
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:05.809075 12191 init.cc:377] Logged in from keytab as oryx/127.4.58.67@KRBTEST.COM (short username oryx)
I20260620 14:31:05.810045 12191 webserver.cc:460] Webserver started at http://127.4.58.67:33879/ using document root <none> and password file <none>
I20260620 14:31:05.810570 12191 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:05.810628 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:05.810796 12191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:05.812934 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "9a9406037ec7465089ad500160a623dc"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "e96938455b57d7b1cab898e6486fb3cd"
server_key_iv: "d2f57dc135d81ead865765da1db6f02b"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.813460 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "9a9406037ec7465089ad500160a623dc"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "e96938455b57d7b1cab898e6486fb3cd"
server_key_iv: "d2f57dc135d81ead865765da1db6f02b"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.817442 12191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:31:05.819911 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.821219 12191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260620 14:31:05.821324 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "9a9406037ec7465089ad500160a623dc"
format_stamp: "Formatted at 2026-06-20 14:31:05 on dist-test-slave-gqzk"
server_key: "e96938455b57d7b1cab898e6486fb3cd"
server_key_iv: "d2f57dc135d81ead865765da1db6f02b"
server_key_version: "encryptionkey@0"
I20260620 14:31:05.821434 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:05.835013 12191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:05.837992 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:05.838168 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:05.839675 12191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:05.839735 12191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:05.839788 12191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:05.839818 12191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:05.850005 12191 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34227
I20260620 14:31:05.850347 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34227 every 8 connection(s)
I20260620 14:31:05.851058 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:05.853161 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.840538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:40489 (local address 127.4.58.126:39097)
0620 14:31:05.840678 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.840683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.841630 (+ 947us) server_negotiation.cc:407] Connection header received
0620 14:31:05.842782 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.842786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.842847 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.842962 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.844298 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.845046 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.846300 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.846522 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.846611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.849445 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.849450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.849453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.849481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.850975 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.851586 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.851609 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.851611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.851705 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.852081 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.852084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.852086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.852271 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.852358 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.852869 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.853000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:31:05.853961 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.840989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.67:40489)
0620 14:31:05.841453 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.841503 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.842530 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.843178 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.843188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.843589 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.844149 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.844164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.845171 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.845175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.846186 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.846195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.846683 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.846688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.846855 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.847436 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.847452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.849267 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.851100 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.851106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.851122 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.851463 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.851830 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.851833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.851835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.851979 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.852371 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.852378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.852625 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.852825 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.853151 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260620 14:31:05.855103 12322 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39097
I20260620 14:31:05.855244 12322 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:05.855729 12322 heartbeater.cc:511] Master 127.4.58.126:39097 requested a full tablet report, sending...
I20260620 14:31:05.856721 11839 ts_manager.cc:194] Registered new tserver with Master: 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227)
I20260620 14:31:05.857437 11839 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:40489
I20260620 14:31:05.859153 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12191
I20260620 14:31:05.859287 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:05.859632 4329 external_mini_cluster.cc:1428] Setting key c343126f717dfd9be092b2cc624599e7
I20260620 14:31:05.861760 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:05.864305 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.857155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.4.58.67:46273)
0620 14:31:05.857424 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.857439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.857547 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.857778 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.857781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.857983 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:05.858103 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.858109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.859107 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.859110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.859901 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.859912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.860338 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.860342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.860448 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.861006 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:05.861028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:05.861410 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:05.862909 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.862912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.862915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.863107 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.863350 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:05.863353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:05.863355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:05.863438 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:05.863841 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:05.863843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:05.863940 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.864031 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.864151 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
I20260620 14:31:05.864362 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.857261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46273 (local address 127.4.58.126:39097)
0620 14:31:05.857372 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.857376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.857509 (+ 133us) server_negotiation.cc:407] Connection header received
0620 14:31:05.857638 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.857640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.857688 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.857754 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.858207 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.858967 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.860026 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.860243 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.860338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.861540 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.861544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.861546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.861572 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.862812 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.863189 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.863193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.863195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.863271 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.863523 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.863526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.863528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.863744 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.863837 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.864093 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.864209 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:05.885248 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:05.879231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.4.58.126:39097)
0620 14:31:05.879379 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.879382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.879394 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:05.879433 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.879435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.879473 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.879575 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.880279 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.880845 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.881680 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.881897 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.881984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.885128 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39786: BlockingRecv error: recv got EOF from 127.0.0.1:39786 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:31:05 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:05.895084 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.886794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.4.58.126:39097)
0620 14:31:05.886909 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.886914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.887110 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:31:05.887215 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.887218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.887258 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.887313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:05.887876 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.888364 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.889208 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.889439 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.889539 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.891746 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:05.891749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:05.891751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:05.891775 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:05.893257 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.893725 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.893727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.893729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.893806 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:05.894182 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:05.894185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:05.894186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:05.894446 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:05.894573 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.894817 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.894923 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:05.898118 11839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:05.901254 11839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:05.915536 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.909622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43083 (local address 127.0.0.1:33560)
0620 14:31:05.909944 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.910007 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.910136 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.911574 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.911577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.911594 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:05.911680 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.911685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.912732 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.912736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.914160 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.914170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.914996 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.915000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.915103 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.915140 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.915251 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.915381 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":88,"thread_start_us":61,"threads_started":1}
I20260620 14:31:05.915546 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.909748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.4.58.65:43083)
0620 14:31:05.911095 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.911110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.911133 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:31:05.911206 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.911211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.911434 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.911589 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:05.911770 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.912615 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.914286 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.914875 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.914976 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.915161 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.915293 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.915390 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":77,"threads_started":1}
I20260620 14:31:05.915705 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.909673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34227 (local address 127.0.0.1:40232)
0620 14:31:05.910358 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.910369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.910445 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.911390 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.911395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.911417 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:05.911531 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.911537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.913305 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.913309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.914382 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.914390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.915341 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.915344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.915429 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.915451 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.915500 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.915568 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":81,"thread_start_us":93,"threads_started":1}
I20260620 14:31:05.915854 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.910353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:34545 (local address 127.0.0.1:48948)
0620 14:31:05.910765 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:05.910777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:05.910857 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:05.911746 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:05.911750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:05.911767 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:05.911853 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.911859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.913188 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.913192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.914255 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:05.914266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.915538 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.915541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:05.915614 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.915634 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:31:05.915693 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:31:05.915750 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260620 14:31:05.916317 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.910464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.4.58.67:34227)
0620 14:31:05.910871 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.910884 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.910909 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:31:05.910984 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.910989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.911180 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.911321 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:05.911657 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.912948 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.914505 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.915220 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.915895 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.916071 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.916113 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.916196 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":90,"threads_started":1}
I20260620 14:31:05.916513 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:05.910749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.4.58.66:34545)
0620 14:31:05.911166 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:05.911175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:05.911192 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:05.911251 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:05.911255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:05.911433 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:05.911589 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:05.911978 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.913071 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.914374 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:05.915197 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:05.915303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:05.915484 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:05.916005 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 14:31:05.916100 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":93,"threads_started":1}
I20260620 14:31:05.918916 12116 tablet_service.cc:1467] Processing CreateTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 (DEFAULT_TABLE table=test-table [id=9db722bc28bc4161a677d59e2f67f2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:05.918941 12256 tablet_service.cc:1467] Processing CreateTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 (DEFAULT_TABLE table=test-table [id=9db722bc28bc4161a677d59e2f67f2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:05.918941 11976 tablet_service.cc:1467] Processing CreateTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 (DEFAULT_TABLE table=test-table [id=9db722bc28bc4161a677d59e2f67f2f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:05.919939 12116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e13a78b9db844c08a790ef9a7010e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:05.919957 11976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e13a78b9db844c08a790ef9a7010e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:05.919963 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e13a78b9db844c08a790ef9a7010e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:05.926940 12350 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Bootstrap starting.
I20260620 14:31:05.928014 12352 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Bootstrap starting.
I20260620 14:31:05.928169 12351 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Bootstrap starting.
I20260620 14:31:05.929651 12350 tablet_bootstrap.cc:654] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:05.929845 12352 tablet_bootstrap.cc:654] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:05.930496 12352 log.cc:824] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:05.930519 12350 log.cc:824] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:05.930613 12351 tablet_bootstrap.cc:654] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:05.931438 12351 log.cc:824] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:05.932682 12350 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: No bootstrap required, opened a new log
I20260620 14:31:05.932857 12352 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: No bootstrap required, opened a new log
I20260620 14:31:05.932912 12350 ts_tablet_manager.cc:1397] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:31:05.933048 12352 ts_tablet_manager.cc:1397] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:31:05.933389 12351 tablet_bootstrap.cc:492] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: No bootstrap required, opened a new log
I20260620 14:31:05.933568 12351 ts_tablet_manager.cc:1397] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 14:31:05.936022 12350 raft_consensus.cc:348] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.936252 12350 raft_consensus.cc:374] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:05.936322 12350 raft_consensus.cc:729] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76547a2fb1f54b1b97715b128b157cd0, State: Initialized, Role: FOLLOWER
I20260620 14:31:05.936790 12350 consensus_queue.cc:260] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.936967 12352 raft_consensus.cc:348] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.937309 12352 raft_consensus.cc:374] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:05.937400 12352 raft_consensus.cc:729] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Initialized, Role: FOLLOWER
I20260620 14:31:05.937526 12182 heartbeater.cc:503] Master 127.4.58.126:39097 was elected leader, sending a full tablet report...
I20260620 14:31:05.937913 12350 ts_tablet_manager.cc:1428] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:31:05.937860 12352 consensus_queue.cc:260] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.937330 12351 raft_consensus.cc:348] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.938822 12352 ts_tablet_manager.cc:1428] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:31:05.938838 12351 raft_consensus.cc:374] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:05.938917 12351 raft_consensus.cc:729] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Initialized, Role: FOLLOWER
I20260620 14:31:05.939205 12322 heartbeater.cc:503] Master 127.4.58.126:39097 was elected leader, sending a full tablet report...
I20260620 14:31:05.939383 12351 consensus_queue.cc:260] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:05.939929 12042 heartbeater.cc:503] Master 127.4.58.126:39097 was elected leader, sending a full tablet report...
I20260620 14:31:05.940842 12351 ts_tablet_manager.cc:1428] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 14:31:05.999229 12357 raft_consensus.cc:482] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:05.999607 12357 raft_consensus.cc:504] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:06.000857 12357 leader_election.cc:288] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.004432 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.001195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:55593 (local address 127.4.58.65:43083)
0620 14:31:06.001325 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.001330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.001569 (+ 239us) server_negotiation.cc:407] Connection header received
0620 14:31:06.001622 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.001624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.001665 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.001730 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:06.001991 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.002505 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.003526 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.003914 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.004008 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.004074 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.004232 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.004307 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:31:06.004428 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.001120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43083 (local address 127.4.58.67:55593)
0620 14:31:06.001323 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.001338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.001491 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.001756 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.001758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.001774 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:06.001876 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.001881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.002611 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.002615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.003402 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.003411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.004027 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.004031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.004125 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.004143 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.004208 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.004283 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 14:31:06.004923 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.001277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:51481 (local address 127.4.58.66:34545)
0620 14:31:06.001382 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.001386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.001628 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:31:06.001765 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.001767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.001811 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.001870 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:06.002485 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.002992 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.003951 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.004326 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.004414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.004473 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.004702 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.004806 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:31:06.005137 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e13a78b9db844c08a790ef9a7010e05" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" is_pre_election: true
I20260620 14:31:06.005263 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.001120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:34545 (local address 127.4.58.67:51481)
0620 14:31:06.001525 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.001536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.001653 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.001892 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.001894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.001910 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:06.002366 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.002373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.003095 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.003098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.003835 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.003844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.004453 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.004457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.004550 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.004576 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.005073 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.005148 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":33,"thread_start_us":108,"threads_started":1}
I20260620 14:31:06.005468 11996 raft_consensus.cc:2454] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.005800 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e13a78b9db844c08a790ef9a7010e05" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0" is_pre_election: true
I20260620 14:31:06.005995 12207 leader_election.cc:302] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.006138 12136 raft_consensus.cc:2454] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.006332 12357 raft_consensus.cc:2781] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:06.006415 12357 raft_consensus.cc:482] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:06.006464 12357 raft_consensus.cc:3037] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.007673 12357 raft_consensus.cc:504] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:06.007994 12357 leader_election.cc:288] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Requested vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.008387 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e13a78b9db844c08a790ef9a7010e05" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
I20260620 14:31:06.008378 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e13a78b9db844c08a790ef9a7010e05" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0"
I20260620 14:31:06.008517 11996 raft_consensus.cc:3037] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.008601 12136 raft_consensus.cc:3037] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.009853 11996 raft_consensus.cc:2454] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.009853 12136 raft_consensus.cc:2454] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.010170 12207 leader_election.cc:302] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.010315 12357 raft_consensus.cc:2781] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:06.010517 12357 raft_consensus.cc:686] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 1 LEADER]: Becoming Leader. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Running, Role: LEADER
I20260620 14:31:06.010885 12357 consensus_queue.cc:237] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } }
I20260620 14:31:06.014148 11836 catalog_manager.cc:5495] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9406037ec7465089ad500160a623dc (127.4.58.67). New cstate: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:06.039979 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.036554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.4.58.67:34227)
0620 14:31:06.036696 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.036701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.036715 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:06.036820 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.036822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.036890 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.036986 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.037231 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.037732 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.038576 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.038797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.038895 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.039148 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.039595 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.039702 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.039768 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.039826 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:31:06.046403 11996 raft_consensus.cc:1261] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.046460 12136 raft_consensus.cc:1261] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.047166 12360 consensus_queue.cc:1035] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.047353 12357 consensus_queue.cc:1035] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.058395 12364 mvcc.cc:205] Tried to move back new op lower bound from 7298932187317776384 to 7298932187183480832. Current Snapshot: MvccSnapshot[applied={T|T < 7298932187317776384}]
I20260620 14:31:06.066080 11836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.066278 11836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.068921 11836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Sending DeleteTablet for 3 replicas of tablet 8e13a78b9db844c08a790ef9a7010e05
I20260620 14:31:06.069789 12116 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48948
I20260620 14:31:06.069837 11976 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33560
I20260620 14:31:06.070286 12373 tablet_replica.cc:330] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: stopping tablet replica
I20260620 14:31:06.070590 12373 raft_consensus.cc:2229] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.070921 12374 tablet_replica.cc:330] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: stopping tablet replica
I20260620 14:31:06.070915 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e13a78b9db844c08a790ef9a7010e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:40232
I20260620 14:31:06.071005 12373 raft_consensus.cc:2258] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.071252 12374 raft_consensus.cc:2229] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.071571 12374 raft_consensus.cc:2258] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.071630 11836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:06.072079 11836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:06.072656 12373 ts_tablet_manager.cc:1905] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.073310 12374 ts_tablet_manager.cc:1905] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.075619 12373 ts_tablet_manager.cc:1918] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.075793 12373 log.cc:1196] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/8e13a78b9db844c08a790ef9a7010e05
I20260620 14:31:06.076130 12374 ts_tablet_manager.cc:1918] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.076237 12373 ts_tablet_manager.cc:1939] T 8e13a78b9db844c08a790ef9a7010e05 P 76547a2fb1f54b1b97715b128b157cd0: Deleting consensus metadata
I20260620 14:31:06.076303 12374 log.cc:1196] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/8e13a78b9db844c08a790ef9a7010e05
I20260620 14:31:06.076680 12374 ts_tablet_manager.cc:1939] T 8e13a78b9db844c08a790ef9a7010e05 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting consensus metadata
I20260620 14:31:06.077476 11976 tablet_service.cc:1467] Processing CreateTablet for tablet fde218a4430e45a19e022cb3c9cee001 (DEFAULT_TABLE table=test-table [id=8d87f719f61846b0b8f7836e78483c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.077477 12256 tablet_service.cc:1467] Processing CreateTablet for tablet fde218a4430e45a19e022cb3c9cee001 (DEFAULT_TABLE table=test-table [id=8d87f719f61846b0b8f7836e78483c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.077729 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde218a4430e45a19e022cb3c9cee001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.077728 11976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde218a4430e45a19e022cb3c9cee001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.079873 12351 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Bootstrap starting.
I20260620 14:31:06.080039 11823 catalog_manager.cc:4841] TS b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083): tablet 8e13a78b9db844c08a790ef9a7010e05 (table test-table [id=9db722bc28bc4161a677d59e2f67f2f2]) successfully deleted
I20260620 14:31:06.080582 12116 tablet_service.cc:1467] Processing CreateTablet for tablet fde218a4430e45a19e022cb3c9cee001 (DEFAULT_TABLE table=test-table [id=8d87f719f61846b0b8f7836e78483c1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.080691 12375 tablet_replica.cc:330] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: stopping tablet replica
I20260620 14:31:06.080864 12116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde218a4430e45a19e022cb3c9cee001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.080920 12351 tablet_bootstrap.cc:654] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.081065 12375 raft_consensus.cc:2229] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:06.081502 12352 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Bootstrap starting.
I20260620 14:31:06.081542 12375 raft_consensus.cc:2258] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.082477 12352 tablet_bootstrap.cc:654] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.083189 12350 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Bootstrap starting.
I20260620 14:31:06.083230 12375 ts_tablet_manager.cc:1905] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.084255 12350 tablet_bootstrap.cc:654] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.085207 12351 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: No bootstrap required, opened a new log
I20260620 14:31:06.085301 12351 ts_tablet_manager.cc:1397] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260620 14:31:06.086007 12351 raft_consensus.cc:348] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.086120 12351 raft_consensus.cc:374] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.086099 12375 ts_tablet_manager.cc:1918] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.086151 12351 raft_consensus.cc:729] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.086185 12375 log.cc:1196] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/8e13a78b9db844c08a790ef9a7010e05
I20260620 14:31:06.086274 12351 consensus_queue.cc:260] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.086513 12375 ts_tablet_manager.cc:1939] T 8e13a78b9db844c08a790ef9a7010e05 P 9a9406037ec7465089ad500160a623dc: Deleting consensus metadata
I20260620 14:31:06.086567 12351 ts_tablet_manager.cc:1428] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.087692 11826 catalog_manager.cc:4841] TS 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227): tablet 8e13a78b9db844c08a790ef9a7010e05 (table test-table [id=9db722bc28bc4161a677d59e2f67f2f2]) successfully deleted
I20260620 14:31:06.087764 11824 catalog_manager.cc:4841] TS 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545): tablet 8e13a78b9db844c08a790ef9a7010e05 (table test-table [id=9db722bc28bc4161a677d59e2f67f2f2]) successfully deleted
I20260620 14:31:06.088704 12350 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: No bootstrap required, opened a new log
I20260620 14:31:06.088928 12352 tablet_bootstrap.cc:492] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: No bootstrap required, opened a new log
I20260620 14:31:06.089007 12352 ts_tablet_manager.cc:1397] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 14:31:06.089000 12350 ts_tablet_manager.cc:1397] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260620 14:31:06.089443 12352 raft_consensus.cc:348] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.089433 12350 raft_consensus.cc:348] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.089546 12352 raft_consensus.cc:374] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.089546 12350 raft_consensus.cc:374] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.089577 12352 raft_consensus.cc:729] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.089577 12350 raft_consensus.cc:729] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76547a2fb1f54b1b97715b128b157cd0, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.089690 12350 consensus_queue.cc:260] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.089690 12352 consensus_queue.cc:260] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.089932 12352 ts_tablet_manager.cc:1428] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:06.089946 12350 ts_tablet_manager.cc:1428] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260620 14:31:06.102044 12323 tablet.cc:2369] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:06.139611 12183 tablet.cc:2369] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:31:06.168639 12043 tablet.cc:2369] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:31:06.249975 12358 raft_consensus.cc:482] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:06.250121 12358 raft_consensus.cc:504] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.251173 12358 leader_election.cc:288] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545), 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227)
I20260620 14:31:06.256373 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.251583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:33045 (local address 127.4.58.67:34227)
0620 14:31:06.251690 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.251694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.251958 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:31:06.252318 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.252320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.252358 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.252411 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:06.252673 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.253411 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255091 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.255690 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255795 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.255884 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.256168 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.256244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:31:06.256373 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.251556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:44651 (local address 127.4.58.66:34545)
0620 14:31:06.251690 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.251694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.251999 (+ 305us) server_negotiation.cc:407] Connection header received
0620 14:31:06.252097 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.252099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.252145 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.252211 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:06.252519 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.253299 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255091 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.255689 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.255883 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.256168 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.256244 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:31:06.256420 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.251460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34227 (local address 127.4.58.65:33045)
0620 14:31:06.251844 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.251859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.252006 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.252448 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.252451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.252468 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:06.252571 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.252576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.253522 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.253525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.254988 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.254996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.255849 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.256014 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.256063 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.256139 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.256230 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":160,"spinlock_wait_cycles":10240,"thread_start_us":107,"threads_started":1}
I20260620 14:31:06.256421 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.251460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:34545 (local address 127.4.58.65:44651)
0620 14:31:06.251844 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.251859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.252007 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.252234 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.252236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.252251 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:06.252359 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.252364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.253426 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.253429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.254968 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.254982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.255848 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.255853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.255996 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.256070 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.256139 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.256230 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":164,"spinlock_wait_cycles":20736,"thread_start_us":97,"threads_started":1}
I20260620 14:31:06.257007 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde218a4430e45a19e022cb3c9cee001" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0" is_pre_election: true
I20260620 14:31:06.257093 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde218a4430e45a19e022cb3c9cee001" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9406037ec7465089ad500160a623dc" is_pre_election: true
I20260620 14:31:06.257184 12136 raft_consensus.cc:2454] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 0.
I20260620 14:31:06.257328 12276 raft_consensus.cc:2454] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 0.
I20260620 14:31:06.257661 11928 leader_election.cc:302] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76547a2fb1f54b1b97715b128b157cd0, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.257918 12358 raft_consensus.cc:2781] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:06.257983 12358 raft_consensus.cc:482] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:06.258028 12358 raft_consensus.cc:3037] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.258963 12358 raft_consensus.cc:504] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.259296 12358 leader_election.cc:288] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 election: Requested vote from peers 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545), 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227)
I20260620 14:31:06.259624 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde218a4430e45a19e022cb3c9cee001" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0"
I20260620 14:31:06.259754 12136 raft_consensus.cc:3037] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.259747 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde218a4430e45a19e022cb3c9cee001" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9406037ec7465089ad500160a623dc"
I20260620 14:31:06.259887 12276 raft_consensus.cc:3037] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.260648 12136 raft_consensus.cc:2454] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 1.
I20260620 14:31:06.260890 12276 raft_consensus.cc:2454] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 1.
I20260620 14:31:06.260946 11928 leader_election.cc:302] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76547a2fb1f54b1b97715b128b157cd0, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.261099 12358 raft_consensus.cc:2781] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:06.261332 12358 raft_consensus.cc:686] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 LEADER]: Becoming Leader. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Running, Role: LEADER
I20260620 14:31:06.261576 12358 consensus_queue.cc:237] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.263739 11839 catalog_manager.cc:5495] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:31:06 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.126@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:06.310760 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.302572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.4.58.126:39097)
0620 14:31:06.302706 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.302711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.302888 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:31:06.303023 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.303026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.303063 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.303128 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:06.303860 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.304368 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.305184 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.305381 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.305466 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.307725 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:06.307728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:06.307731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:06.307756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:06.309050 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:06.309557 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:06.309560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:06.309561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:06.309629 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:06.309857 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:06.309860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:06.309862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:06.310066 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:06.310152 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.310423 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.310605 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":50}
I20260620 14:31:06.310760 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.302487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39097 (local address 127.0.0.1:39806)
0620 14:31:06.302793 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.302805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.302926 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.303159 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.303161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.303636 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:06.303733 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.303741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.304469 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.304473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.305071 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.305080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.305479 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.305482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.305598 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.306110 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:06.306122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:06.307591 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:06.309140 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:06.309151 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:06.309158 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:06.309466 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:06.309704 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:06.309707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:06.309709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:06.309785 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:06.310157 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:06.310165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:06.310273 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.310368 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.310538 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":31,"thread_start_us":116,"threads_started":1}
I20260620 14:31:06.315117 11839 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:39806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:31:06.315796 11839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:06.320415 11976 tablet_service.cc:1467] Processing CreateTablet for tablet 09d95cfa197047e0aa893a0403015d29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=974d7c3a4a294c04b5eb703f15ef78cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:06.320508 12116 tablet_service.cc:1467] Processing CreateTablet for tablet 09d95cfa197047e0aa893a0403015d29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=974d7c3a4a294c04b5eb703f15ef78cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:06.320624 12256 tablet_service.cc:1467] Processing CreateTablet for tablet 09d95cfa197047e0aa893a0403015d29 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=974d7c3a4a294c04b5eb703f15ef78cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:31:06.320906 11976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d95cfa197047e0aa893a0403015d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.320955 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d95cfa197047e0aa893a0403015d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.320947 12116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d95cfa197047e0aa893a0403015d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.323660 12352 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: Bootstrap starting.
I20260620 14:31:06.323674 12350 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: Bootstrap starting.
I20260620 14:31:06.323717 12351 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: Bootstrap starting.
I20260620 14:31:06.324538 12351 tablet_bootstrap.cc:654] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.324579 12350 tablet_bootstrap.cc:654] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.324579 12352 tablet_bootstrap.cc:654] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.325827 12350 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: No bootstrap required, opened a new log
I20260620 14:31:06.325855 12352 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: No bootstrap required, opened a new log
I20260620 14:31:06.325918 12352 ts_tablet_manager.cc:1397] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:06.325918 12350 ts_tablet_manager.cc:1397] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:31:06.326339 12352 raft_consensus.cc:348] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.326339 12350 raft_consensus.cc:348] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.326467 12352 raft_consensus.cc:374] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.326467 12350 raft_consensus.cc:374] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.326506 12352 raft_consensus.cc:729] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.326506 12350 raft_consensus.cc:729] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76547a2fb1f54b1b97715b128b157cd0, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.326628 12352 consensus_queue.cc:260] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.326628 12350 consensus_queue.cc:260] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.326893 12350 ts_tablet_manager.cc:1428] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.326859 12369 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.326859 12356 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.327013 12369 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.327013 12356 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.326884 12352 ts_tablet_manager.cc:1428] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.327656 12351 tablet_bootstrap.cc:492] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: No bootstrap required, opened a new log
I20260620 14:31:06.327746 12351 ts_tablet_manager.cc:1397] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 14:31:06.328186 12351 raft_consensus.cc:348] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.328296 12351 raft_consensus.cc:374] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.328325 12351 raft_consensus.cc:729] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.328449 12351 consensus_queue.cc:260] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.328672 12358 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.328852 12358 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.328692 12351 ts_tablet_manager.cc:1428] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.365976 12369 raft_consensus.cc:482] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:06.366112 12369 raft_consensus.cc:504] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.366539 12369 leader_election.cc:288] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.367057 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d95cfa197047e0aa893a0403015d29" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" is_pre_election: true
I20260620 14:31:06.367043 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d95cfa197047e0aa893a0403015d29" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0" is_pre_election: true
I20260620 14:31:06.367206 11996 raft_consensus.cc:2454] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.367226 12136 raft_consensus.cc:2454] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.367537 12208 leader_election.cc:302] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 76547a2fb1f54b1b97715b128b157cd0, 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.367699 12369 raft_consensus.cc:2781] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:06.367758 12369 raft_consensus.cc:482] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:06.367791 12369 raft_consensus.cc:3037] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.368744 12369 raft_consensus.cc:504] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.369100 12369 leader_election.cc:288] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Requested vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.369433 12136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d95cfa197047e0aa893a0403015d29" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0"
I20260620 14:31:06.369436 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d95cfa197047e0aa893a0403015d29" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
I20260620 14:31:06.369553 11996 raft_consensus.cc:3037] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.369553 12136 raft_consensus.cc:3037] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.370673 11996 raft_consensus.cc:2454] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.370672 12136 raft_consensus.cc:2454] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.370944 12207 leader_election.cc:302] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.371109 12369 raft_consensus.cc:2781] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:06.371196 12369 raft_consensus.cc:686] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [term 1 LEADER]: Becoming Leader. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Running, Role: LEADER
I20260620 14:31:06.371331 12369 consensus_queue.cc:237] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.372008 12357 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: TxnStatusTablet state changed. Reason: New leader 9a9406037ec7465089ad500160a623dc. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.372114 12357 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:06.372316 12393 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:31:06.372475 12393 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:31:06.373123 11836 catalog_manager.cc:5495] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9406037ec7465089ad500160a623dc (127.4.58.67). New cstate: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:06.392261 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.388659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.4.58.67:34227)
0620 14:31:06.388801 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.388806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.388893 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:31:06.389059 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.389061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.389108 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.389177 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.389498 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.390011 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.391217 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.391485 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.391573 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.391858 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.391930 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.391978 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.392085 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.392133 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:31:06.392304 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.388555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34227 (local address 127.0.0.1:40258)
0620 14:31:06.388823 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.388838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.388955 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.389205 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.389208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.389215 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:06.389323 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.389329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.390125 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.390128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.391089 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.391099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.391588 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.391592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.391683 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.391740 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.391999 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.392010 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.392068 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.392161 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 14:31:06.394945 11996 raft_consensus.cc:1261] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.394945 12136 raft_consensus.cc:1261] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.395375 12369 consensus_queue.cc:1035] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.395530 12357 consensus_queue.cc:1035] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.396907 12358 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: TxnStatusTablet state changed. Reason: New leader 9a9406037ec7465089ad500160a623dc. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.397032 12358 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.398098 12356 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: TxnStatusTablet state changed. Reason: New leader 9a9406037ec7465089ad500160a623dc. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.398213 12356 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.399333 12369 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.399453 12369 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:06.399565 12369 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.399638 12369 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 9a9406037ec7465089ad500160a623dc: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:31:06.399720 12363 mvcc.cc:205] Tried to move back new op lower bound from 7298932188749606912 to 7298932188658413568. Current Snapshot: MvccSnapshot[applied={T|T < 7298932188749606912 or (T in {7298932188749606912})}]
I20260620 14:31:06.399756 12365 mvcc.cc:205] Tried to move back new op lower bound from 7298932188749606912 to 7298932188658413568. Current Snapshot: MvccSnapshot[applied={T|T < 7298932188749606912}]
I20260620 14:31:06.400226 12356 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.400317 12356 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P 76547a2fb1f54b1b97715b128b157cd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.402709 12358 tablet_replica.cc:439] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } } }
I20260620 14:31:06.402824 12358 tablet_replica.cc:442] T 09d95cfa197047e0aa893a0403015d29 P b8f5b29fe05046d5b2dc2ca88d898036: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:31:06.408825 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.404580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.4.58.65:43083)
0620 14:31:06.404854 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.404858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.404933 (+ 75us) server_negotiation.cc:407] Connection header received
0620 14:31:06.405105 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.405108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.405171 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.405279 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.405485 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.406250 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.407086 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.407435 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.407534 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.407795 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.408387 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.408513 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.408605 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.408664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260620 14:31:06.421500 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.416991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:39007 (local address 127.4.58.67:34227)
0620 14:31:06.417128 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.417133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.417309 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:31:06.417444 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.417448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.417504 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.417577 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.418219 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.419000 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.420271 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.420625 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.420728 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.421061 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.421152 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.421219 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.421293 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.421352 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:06.421527 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.416787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34227 (local address 127.4.58.65:39007)
0620 14:31:06.417224 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.417239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.417347 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.417600 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.417602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.417611 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:06.418108 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.418118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.419104 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.419107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.420149 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.420159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.420739 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.420743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.420874 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.420938 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.421227 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.421232 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.421285 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.421363 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":93,"threads_started":1}
I20260620 14:31:06.430846 12276 raft_consensus.cc:1261] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Refusing update from remote peer b8f5b29fe05046d5b2dc2ca88d898036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.431305 12136 raft_consensus.cc:1261] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Refusing update from remote peer b8f5b29fe05046d5b2dc2ca88d898036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.431479 12381 consensus_queue.cc:1035] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.432122 12381 consensus_queue.cc:1035] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.456293 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.451818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:48083 (local address 127.4.58.65:43083)
0620 14:31:06.451934 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.451939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.452120 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:31:06.452264 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.452267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.452324 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.452414 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.452887 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.453872 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.455091 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.455406 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.455501 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.455802 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.455888 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.455958 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.456073 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.456153 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:06.456704 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.451749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43083 (local address 127.4.58.67:48083)
0620 14:31:06.452033 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.452046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:06.452145 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:06.452526 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:06.452529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:06.452539 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:31:06.452636 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.452642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.453976 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.453980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.454958 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:06.454969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.455517 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.455521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:06.455608 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.455665 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.455965 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.455972 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:06.456481 (+ 509us) client_negotiation.cc:240] Negotiation successful
0620 14:31:06.456569 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260620 14:31:06.484272 11836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.484467 11836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.487041 11836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Sending DeleteTablet for 3 replicas of tablet fde218a4430e45a19e022cb3c9cee001
I20260620 14:31:06.487715 11976 tablet_service.cc:1514] Processing DeleteTablet for tablet fde218a4430e45a19e022cb3c9cee001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33560
I20260620 14:31:06.487847 12116 tablet_service.cc:1514] Processing DeleteTablet for tablet fde218a4430e45a19e022cb3c9cee001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48948
I20260620 14:31:06.487928 12374 tablet_replica.cc:330] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: stopping tablet replica
I20260620 14:31:06.488040 12374 raft_consensus.cc:2229] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:06.488371 12374 raft_consensus.cc:2258] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.488448 12373 tablet_replica.cc:330] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: stopping tablet replica
I20260620 14:31:06.488569 12373 raft_consensus.cc:2229] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.488689 12373 raft_consensus.cc:2258] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.489566 12373 ts_tablet_manager.cc:1905] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.491220 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet fde218a4430e45a19e022cb3c9cee001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:40232
I20260620 14:31:06.491421 12375 tablet_replica.cc:330] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: stopping tablet replica
I20260620 14:31:06.491531 12375 raft_consensus.cc:2229] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.491648 12375 raft_consensus.cc:2258] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.492489 12375 ts_tablet_manager.cc:1905] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.492614 12373 ts_tablet_manager.cc:1918] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:06.492672 12373 log.cc:1196] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/fde218a4430e45a19e022cb3c9cee001
I20260620 14:31:06.492928 12373 ts_tablet_manager.cc:1939] T fde218a4430e45a19e022cb3c9cee001 P 76547a2fb1f54b1b97715b128b157cd0: Deleting consensus metadata
I20260620 14:31:06.493701 11824 catalog_manager.cc:4841] TS 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545): tablet fde218a4430e45a19e022cb3c9cee001 (table test-table [id=8d87f719f61846b0b8f7836e78483c1d]) successfully deleted
I20260620 14:31:06.494721 12374 ts_tablet_manager.cc:1905] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.496208 12375 ts_tablet_manager.cc:1918] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:06.496286 12375 log.cc:1196] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/fde218a4430e45a19e022cb3c9cee001
I20260620 14:31:06.496518 12375 ts_tablet_manager.cc:1939] T fde218a4430e45a19e022cb3c9cee001 P 9a9406037ec7465089ad500160a623dc: Deleting consensus metadata
I20260620 14:31:06.497329 11826 catalog_manager.cc:4841] TS 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227): tablet fde218a4430e45a19e022cb3c9cee001 (table test-table [id=8d87f719f61846b0b8f7836e78483c1d]) successfully deleted
I20260620 14:31:06.497503 12374 ts_tablet_manager.cc:1918] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:06.497568 12374 log.cc:1196] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/fde218a4430e45a19e022cb3c9cee001
I20260620 14:31:06.497747 12374 ts_tablet_manager.cc:1939] T fde218a4430e45a19e022cb3c9cee001 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting consensus metadata
I20260620 14:31:06.498486 11823 catalog_manager.cc:4841] TS b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083): tablet fde218a4430e45a19e022cb3c9cee001 (table test-table [id=8d87f719f61846b0b8f7836e78483c1d]) successfully deleted
Jun 20 14:31:06 dist-test-slave-gqzk krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:31:06 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:06 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:06.520541 11888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:06.514518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.4.58.126:39097)
0620 14:31:06.514645 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.514650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.514961 (+ 311us) server_negotiation.cc:407] Connection header received
0620 14:31:06.515085 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.515088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.515132 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.515199 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:06.515925 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.516442 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.517352 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.517570 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.517660 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.520412 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39812: BlockingRecv error: recv got EOF from 127.0.0.1:39812 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:31:06 dist-test-slave-gqzk krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM
I20260620 14:31:06.530457 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.522638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.4.58.126:39097)
0620 14:31:06.522740 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.522744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.522773 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:31:06.522906 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.522909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.522955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.523023 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:06.523697 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.524214 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.525099 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.525311 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.525466 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.527625 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:06.527628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:06.527630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:06.527656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:06.528980 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:06.529349 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:06.529353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:06.529355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:06.529433 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:06.529668 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:06.529671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:06.529673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:06.529857 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:06.529972 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.530185 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.530290 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:06.533191 11836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:06.533638 11836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:06.538075 11976 tablet_service.cc:1467] Processing CreateTablet for tablet b63e5014624f4a7aa7559e4f15334d01 (DEFAULT_TABLE table=test-table [id=8132d1e5e3dd40bcb133419b179eed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.538281 12116 tablet_service.cc:1467] Processing CreateTablet for tablet b63e5014624f4a7aa7559e4f15334d01 (DEFAULT_TABLE table=test-table [id=8132d1e5e3dd40bcb133419b179eed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.538408 11976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63e5014624f4a7aa7559e4f15334d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.538512 12116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63e5014624f4a7aa7559e4f15334d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.540803 12351 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Bootstrap starting.
I20260620 14:31:06.540803 12350 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Bootstrap starting.
I20260620 14:31:06.541730 12350 tablet_bootstrap.cc:654] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.541730 12351 tablet_bootstrap.cc:654] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.543105 12256 tablet_service.cc:1467] Processing CreateTablet for tablet b63e5014624f4a7aa7559e4f15334d01 (DEFAULT_TABLE table=test-table [id=8132d1e5e3dd40bcb133419b179eed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.543378 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b63e5014624f4a7aa7559e4f15334d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.544097 12350 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: No bootstrap required, opened a new log
I20260620 14:31:06.544170 12350 ts_tablet_manager.cc:1397] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:06.544631 12350 raft_consensus.cc:348] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.544741 12350 raft_consensus.cc:374] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.544793 12350 raft_consensus.cc:729] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76547a2fb1f54b1b97715b128b157cd0, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.544914 12350 consensus_queue.cc:260] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.545190 12350 ts_tablet_manager.cc:1428] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:31:06.545965 12352 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Bootstrap starting.
I20260620 14:31:06.546746 12352 tablet_bootstrap.cc:654] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.546880 12351 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: No bootstrap required, opened a new log
I20260620 14:31:06.546968 12351 ts_tablet_manager.cc:1397] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:31:06.547433 12351 raft_consensus.cc:348] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.547544 12351 raft_consensus.cc:374] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.547574 12351 raft_consensus.cc:729] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.547658 12352 tablet_bootstrap.cc:492] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: No bootstrap required, opened a new log
I20260620 14:31:06.547729 12352 ts_tablet_manager.cc:1397] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:06.547695 12351 consensus_queue.cc:260] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.547940 12351 ts_tablet_manager.cc:1428] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.548079 12352 raft_consensus.cc:348] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.548177 12352 raft_consensus.cc:374] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.548204 12352 raft_consensus.cc:729] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.548314 12352 consensus_queue.cc:260] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.548571 12352 ts_tablet_manager.cc:1428] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:06.657876 12357 raft_consensus.cc:482] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:06.658018 12357 raft_consensus.cc:504] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.658388 12357 leader_election.cc:288] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.658872 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63e5014624f4a7aa7559e4f15334d01" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0" is_pre_election: true
I20260620 14:31:06.658872 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63e5014624f4a7aa7559e4f15334d01" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" is_pre_election: true
I20260620 14:31:06.659036 11996 raft_consensus.cc:2454] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.659034 12135 raft_consensus.cc:2454] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 0.
I20260620 14:31:06.659355 12207 leader_election.cc:302] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.659561 12357 raft_consensus.cc:2781] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:06.659616 12357 raft_consensus.cc:482] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:06.659643 12357 raft_consensus.cc:3037] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.660610 12357 raft_consensus.cc:504] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.660976 12357 leader_election.cc:288] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Requested vote from peers b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083), 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545)
I20260620 14:31:06.661319 11996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63e5014624f4a7aa7559e4f15334d01" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5b29fe05046d5b2dc2ca88d898036"
I20260620 14:31:06.661346 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63e5014624f4a7aa7559e4f15334d01" candidate_uuid: "9a9406037ec7465089ad500160a623dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0"
I20260620 14:31:06.661433 11996 raft_consensus.cc:3037] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.661463 12135 raft_consensus.cc:3037] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.662289 11996 raft_consensus.cc:2454] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.662336 12135 raft_consensus.cc:2454] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9406037ec7465089ad500160a623dc in term 1.
I20260620 14:31:06.662588 12207 leader_election.cc:302] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.662770 12357 raft_consensus.cc:2781] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:06.662843 12357 raft_consensus.cc:686] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 1 LEADER]: Becoming Leader. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Running, Role: LEADER
I20260620 14:31:06.662989 12357 consensus_queue.cc:237] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.664669 11836 catalog_manager.cc:5495] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9406037ec7465089ad500160a623dc (127.4.58.67). New cstate: current_term: 1 leader_uuid: "9a9406037ec7465089ad500160a623dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:06.702769 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.700055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.4.58.67:34227)
0620 14:31:06.700162 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.700165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.700177 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:06.700257 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.700260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.700302 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.700367 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.700557 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.701092 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.701934 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.702145 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.702223 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.702404 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.702482 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.702537 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.702611 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.702658 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:31:06.704680 12135 raft_consensus.cc:1261] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.704789 11996 raft_consensus.cc:1261] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Refusing update from remote peer 9a9406037ec7465089ad500160a623dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.705194 12360 consensus_queue.cc:1035] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.705358 12357 consensus_queue.cc:1035] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.712276 11836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.712404 11836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.714183 11836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Sending DeleteTablet for 3 replicas of tablet b63e5014624f4a7aa7559e4f15334d01
I20260620 14:31:06.714665 11976 tablet_service.cc:1514] Processing DeleteTablet for tablet b63e5014624f4a7aa7559e4f15334d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33560
I20260620 14:31:06.714676 12116 tablet_service.cc:1514] Processing DeleteTablet for tablet b63e5014624f4a7aa7559e4f15334d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48948
I20260620 14:31:06.714712 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet b63e5014624f4a7aa7559e4f15334d01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:40232
I20260620 14:31:06.714840 12375 tablet_replica.cc:330] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: stopping tablet replica
I20260620 14:31:06.714834 12373 tablet_replica.cc:330] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: stopping tablet replica
I20260620 14:31:06.714864 12374 tablet_replica.cc:330] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: stopping tablet replica
I20260620 14:31:06.714932 12373 raft_consensus.cc:2229] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.714932 12375 raft_consensus.cc:2229] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:06.714993 12374 raft_consensus.cc:2229] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.715040 12373 raft_consensus.cc:2258] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.715109 12375 raft_consensus.cc:2258] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.715118 12374 raft_consensus.cc:2258] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.715435 11836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:06.715811 11836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:06.715852 12373 ts_tablet_manager.cc:1905] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.715889 12374 ts_tablet_manager.cc:1905] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.718079 12374 ts_tablet_manager.cc:1918] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.718156 12374 log.cc:1196] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/b63e5014624f4a7aa7559e4f15334d01
I20260620 14:31:06.718196 12373 ts_tablet_manager.cc:1918] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.718266 12373 log.cc:1196] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/b63e5014624f4a7aa7559e4f15334d01
I20260620 14:31:06.718384 12374 ts_tablet_manager.cc:1939] T b63e5014624f4a7aa7559e4f15334d01 P b8f5b29fe05046d5b2dc2ca88d898036: Deleting consensus metadata
I20260620 14:31:06.718530 12373 ts_tablet_manager.cc:1939] T b63e5014624f4a7aa7559e4f15334d01 P 76547a2fb1f54b1b97715b128b157cd0: Deleting consensus metadata
I20260620 14:31:06.719063 11823 catalog_manager.cc:4841] TS b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083): tablet b63e5014624f4a7aa7559e4f15334d01 (table test-table [id=8132d1e5e3dd40bcb133419b179eed00]) successfully deleted
I20260620 14:31:06.719175 11824 catalog_manager.cc:4841] TS 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545): tablet b63e5014624f4a7aa7559e4f15334d01 (table test-table [id=8132d1e5e3dd40bcb133419b179eed00]) successfully deleted
I20260620 14:31:06.719403 12375 ts_tablet_manager.cc:1905] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.720359 11976 tablet_service.cc:1467] Processing CreateTablet for tablet e9e57a1545a044c7b8a648096d17881a (DEFAULT_TABLE table=test-table [id=23507c02199644cd937b604d377ab46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.720518 12256 tablet_service.cc:1467] Processing CreateTablet for tablet e9e57a1545a044c7b8a648096d17881a (DEFAULT_TABLE table=test-table [id=23507c02199644cd937b604d377ab46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.720613 11976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e57a1545a044c7b8a648096d17881a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.720818 12256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e57a1545a044c7b8a648096d17881a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.721783 12375 ts_tablet_manager.cc:1918] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:06.721858 12375 log.cc:1196] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/b63e5014624f4a7aa7559e4f15334d01
I20260620 14:31:06.722079 12375 ts_tablet_manager.cc:1939] T b63e5014624f4a7aa7559e4f15334d01 P 9a9406037ec7465089ad500160a623dc: Deleting consensus metadata
I20260620 14:31:06.722736 12352 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Bootstrap starting.
I20260620 14:31:06.722798 12351 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: Bootstrap starting.
I20260620 14:31:06.722826 12116 tablet_service.cc:1467] Processing CreateTablet for tablet e9e57a1545a044c7b8a648096d17881a (DEFAULT_TABLE table=test-table [id=23507c02199644cd937b604d377ab46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:06.722952 11826 catalog_manager.cc:4841] TS 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227): tablet b63e5014624f4a7aa7559e4f15334d01 (table test-table [id=8132d1e5e3dd40bcb133419b179eed00]) successfully deleted
I20260620 14:31:06.723052 12116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e57a1545a044c7b8a648096d17881a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:06.723480 12352 tablet_bootstrap.cc:654] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.723558 12351 tablet_bootstrap.cc:654] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.724490 12352 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: No bootstrap required, opened a new log
I20260620 14:31:06.724565 12352 ts_tablet_manager.cc:1397] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:31:06.724963 12352 raft_consensus.cc:348] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.725080 12352 raft_consensus.cc:374] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.725111 12352 raft_consensus.cc:729] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9406037ec7465089ad500160a623dc, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.725162 12350 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Bootstrap starting.
I20260620 14:31:06.725227 12352 consensus_queue.cc:260] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.725477 12352 ts_tablet_manager.cc:1428] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:06.725745 12351 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: No bootstrap required, opened a new log
I20260620 14:31:06.725819 12351 ts_tablet_manager.cc:1397] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:06.726079 12350 tablet_bootstrap.cc:654] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:06.726167 12351 raft_consensus.cc:348] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.726274 12351 raft_consensus.cc:374] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.726315 12351 raft_consensus.cc:729] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.726430 12351 consensus_queue.cc:260] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.726678 12351 ts_tablet_manager.cc:1428] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:31:06.727397 12350 tablet_bootstrap.cc:492] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: No bootstrap required, opened a new log
I20260620 14:31:06.727478 12350 ts_tablet_manager.cc:1397] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:31:06.727893 12350 raft_consensus.cc:348] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.727998 12350 raft_consensus.cc:374] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:06.728029 12350 raft_consensus.cc:729] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76547a2fb1f54b1b97715b128b157cd0, State: Initialized, Role: FOLLOWER
I20260620 14:31:06.728144 12350 consensus_queue.cc:260] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.728365 12350 ts_tablet_manager.cc:1428] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:31:06.899281 12405 raft_consensus.cc:482] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:06.899420 12405 raft_consensus.cc:504] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.899842 12405 leader_election.cc:288] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545), 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227)
I20260620 14:31:06.900342 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e57a1545a044c7b8a648096d17881a" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0" is_pre_election: true
I20260620 14:31:06.900341 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e57a1545a044c7b8a648096d17881a" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9406037ec7465089ad500160a623dc" is_pre_election: true
I20260620 14:31:06.900496 12276 raft_consensus.cc:2454] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 0.
I20260620 14:31:06.900529 12135 raft_consensus.cc:2454] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 0.
I20260620 14:31:06.900812 11930 leader_election.cc:302] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a9406037ec7465089ad500160a623dc, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.900990 12405 raft_consensus.cc:2781] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:06.901052 12405 raft_consensus.cc:482] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:06.901078 12405 raft_consensus.cc:3037] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.902093 12405 raft_consensus.cc:504] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.902426 12405 leader_election.cc:288] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 election: Requested vote from peers 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545), 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227)
I20260620 14:31:06.902796 12276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e57a1545a044c7b8a648096d17881a" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9406037ec7465089ad500160a623dc"
I20260620 14:31:06.902792 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e57a1545a044c7b8a648096d17881a" candidate_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76547a2fb1f54b1b97715b128b157cd0"
I20260620 14:31:06.902917 12276 raft_consensus.cc:3037] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.902923 12135 raft_consensus.cc:3037] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:06.903796 12276 raft_consensus.cc:2454] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 1.
I20260620 14:31:06.903898 12135 raft_consensus.cc:2454] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5b29fe05046d5b2dc2ca88d898036 in term 1.
I20260620 14:31:06.904172 11928 leader_election.cc:302] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76547a2fb1f54b1b97715b128b157cd0, b8f5b29fe05046d5b2dc2ca88d898036; no voters:
I20260620 14:31:06.904340 12405 raft_consensus.cc:2781] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:06.904433 12405 raft_consensus.cc:686] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 LEADER]: Becoming Leader. State: Replica: b8f5b29fe05046d5b2dc2ca88d898036, State: Running, Role: LEADER
I20260620 14:31:06.904569 12405 consensus_queue.cc:237] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } }
I20260620 14:31:06.906194 11838 catalog_manager.cc:5495] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5b29fe05046d5b2dc2ca88d898036" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 } health_report { overall_health: UNKNOWN } } }
I20260620 14:31:06.924543 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.921519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.4.58.65:43083)
0620 14:31:06.921619 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:06.921623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:06.921680 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:31:06.921801 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:06.921804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:06.921849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:06.921929 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:06.922162 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.922748 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.923590 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:06.923803 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:06.923891 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:06.924113 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:06.924185 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:06.924253 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:06.924334 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:31:06.924434 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:31:06.930392 12276 raft_consensus.cc:1261] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Refusing update from remote peer b8f5b29fe05046d5b2dc2ca88d898036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.930428 12135 raft_consensus.cc:1261] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Refusing update from remote peer b8f5b29fe05046d5b2dc2ca88d898036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:06.930761 12405 consensus_queue.cc:1035] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9406037ec7465089ad500160a623dc" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.930934 12358 consensus_queue.cc:1035] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76547a2fb1f54b1b97715b128b157cd0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:06.957414 11838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.957566 11838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:06.959291 11838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2f3f17cc05a4baa9d212a8f99804dea: Sending DeleteTablet for 3 replicas of tablet e9e57a1545a044c7b8a648096d17881a
I20260620 14:31:06.959797 11976 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e57a1545a044c7b8a648096d17881a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33560
I20260620 14:31:06.959889 12256 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e57a1545a044c7b8a648096d17881a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:40232
I20260620 14:31:06.959980 12374 tablet_replica.cc:330] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036: stopping tablet replica
I20260620 14:31:06.959982 12116 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e57a1545a044c7b8a648096d17881a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:06 UTC) from {username='oryx', principal='oryx/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48948
I20260620 14:31:06.960057 12375 tablet_replica.cc:330] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: stopping tablet replica
I20260620 14:31:06.960093 12374 raft_consensus.cc:2229] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:31:06.960132 12373 tablet_replica.cc:330] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: stopping tablet replica
I20260620 14:31:06.960145 12375 raft_consensus.cc:2229] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.960223 12373 raft_consensus.cc:2229] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:06.960240 12375 raft_consensus.cc:2258] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.960266 12374 raft_consensus.cc:2258] T e9e57a1545a044c7b8a648096d17881a P b8f5b29fe05046d5b2dc2ca88d898036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.960319 12373 raft_consensus.cc:2258] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:06.960585 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 11911
I20260620 14:31:06.960947 12375 ts_tablet_manager.cc:1905] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.962000 12373 ts_tablet_manager.cc:1905] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:06.963076 12375 ts_tablet_manager.cc:1918] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:06.963143 12375 log.cc:1196] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/e9e57a1545a044c7b8a648096d17881a
I20260620 14:31:06.963330 12375 ts_tablet_manager.cc:1939] T e9e57a1545a044c7b8a648096d17881a P 9a9406037ec7465089ad500160a623dc: Deleting consensus metadata
I20260620 14:31:06.964134 12373 ts_tablet_manager.cc:1918] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:31:06.964197 12373 log.cc:1196] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/e9e57a1545a044c7b8a648096d17881a
I20260620 14:31:06.964277 11826 catalog_manager.cc:4841] TS 9a9406037ec7465089ad500160a623dc (127.4.58.67:34227): tablet e9e57a1545a044c7b8a648096d17881a (table test-table [id=23507c02199644cd937b604d377ab46f]) successfully deleted
I20260620 14:31:06.964409 12373 ts_tablet_manager.cc:1939] T e9e57a1545a044c7b8a648096d17881a P 76547a2fb1f54b1b97715b128b157cd0: Deleting consensus metadata
I20260620 14:31:06.965091 11824 catalog_manager.cc:4841] TS 76547a2fb1f54b1b97715b128b157cd0 (127.4.58.66:34545): tablet e9e57a1545a044c7b8a648096d17881a (table test-table [id=23507c02199644cd937b604d377ab46f]) successfully deleted
W20260620 14:31:06.969267 11823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:43083 (error 108)
I20260620 14:31:06.970081 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12051
I20260620 14:31:06.971385 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:06.969951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43083 (local address 127.0.0.1:33598)
0620 14:31:06.971189 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:06.971278 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:43083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
W20260620 14:31:06.971622 11823 catalog_manager.cc:4584] TS b8f5b29fe05046d5b2dc2ca88d898036 (127.4.58.65:43083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9e57a1545a044c7b8a648096d17881a: Network error: Client connection negotiation failed: client connection to 127.4.58.65:43083: connect: Connection refused (error 111)
I20260620 14:31:06.975368 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12191
I20260620 14:31:06.980921 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 11807
2026-06-20T14:31:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3102 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:07 dist-test-slave-gqzk krb5kdc[12437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:07 dist-test-slave-gqzk krb5kdc[12437](info): set up 2 sockets
Jun 20 14:31:07 dist-test-slave-gqzk krb5kdc[12437](info): commencing operation
krb5kdc: starting...
W20260620 14:31:07.937273 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:31:08 dist-test-slave-gqzk krb5kdc[12437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:31:08.062037 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38861
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:36071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:38861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:08.172318 12450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:08.172608 12450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:08.172649 12450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:08.176637 12450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:08.176708 12450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:08.176728 12450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:08.176743 12450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:08.176795 12450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:08.182153 12450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:38861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:08.183183 12450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:08.184180 12450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:08.192778 12455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:08.192836 12458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:08.192852 12450 server_base.cc:1034] running on GCE node
W20260620 14:31:08.192975 12456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:08.193629 12450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:08.194767 12450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:08.195981 12450 hybrid_clock.cc:648] HybridClock initialized: now 1781965868195952 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:31:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1219 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:08 dist-test-slave-gqzk krb5kdc[12463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:08 dist-test-slave-gqzk krb5kdc[12463](info): set up 2 sockets
Jun 20 14:31:08 dist-test-slave-gqzk krb5kdc[12463](info): commencing operation
krb5kdc: starting...
W20260620 14:31:09.157246 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:31:09.275067 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44693
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:09.384847 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:09.385139 12476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:09.385180 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:09.389206 12476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:09.389271 12476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:09.389288 12476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:09.389304 12476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:09.389317 12476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:09.394697 12476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:09.395823 12476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:09.396945 12476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:09.405385 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:09.405467 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.405529 12476 server_base.cc:1034] running on GCE node
W20260620 14:31:09.405447 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.406239 12476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:09.407414 12476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:09.408586 12476 hybrid_clock.cc:648] HybridClock initialized: now 1781965869408567 us; error 29 us; skew 500 ppm
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:09.415987 12476 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:31:09.416996 12476 webserver.cc:460] Webserver started at http://127.4.58.126:43511/ using document root <none> and password file <none>
I20260620 14:31:09.417649 12476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:09.417725 12476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:09.417944 12476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:09.420042 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "0ba85fc53b42405b9683bfeb236dfe35"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "4b11ef64ab394f5d1a6630be6d8de68a"
server_key_iv: "28ebf78b28f5239407ad1f8afb122b09"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.420563 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "0ba85fc53b42405b9683bfeb236dfe35"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "4b11ef64ab394f5d1a6630be6d8de68a"
server_key_iv: "28ebf78b28f5239407ad1f8afb122b09"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.424260 12476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:31:09.426810 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.428160 12476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:09.428267 12476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "0ba85fc53b42405b9683bfeb236dfe35"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "4b11ef64ab394f5d1a6630be6d8de68a"
server_key_iv: "28ebf78b28f5239407ad1f8afb122b09"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.428375 12476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:09.444696 12476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:09.447885 12476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:09.448057 12476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:09.455183 12476 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44693
I20260620 14:31:09.455204 12543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44693 every 8 connection(s)
I20260620 14:31:09.456122 12476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:09.459004 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:09.461421 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12476
I20260620 14:31:09.461524 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:31:09.461787 4329 external_mini_cluster.cc:1428] Setting key 613bc54e81136577304c1a9447a7cca0
I20260620 14:31:09.464932 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Bootstrap starting.
I20260620 14:31:09.467116 12544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:09.467852 12544 log.cc:824] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:09.469719 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: No bootstrap required, opened a new log
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:09.473058 12544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:09.473284 12544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:09.473335 12544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba85fc53b42405b9683bfeb236dfe35, State: Initialized, Role: FOLLOWER
I20260620 14:31:09.473788 12544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:09.473920 12544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:09.473989 12544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:09.474098 12544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:09.475234 12544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:09.475611 12544 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba85fc53b42405b9683bfeb236dfe35; no voters:
I20260620 14:31:09.475947 12544 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:09.476082 12549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:09.476271 12549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba85fc53b42405b9683bfeb236dfe35, State: Running, Role: LEADER
I20260620 14:31:09.476315 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:09.463176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.4.58.126:44693)
0620 14:31:09.463602 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:09.463617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:09.463653 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:31:09.464277 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:09.464295 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:09.464638 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:09.464993 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:09.465826 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.466813 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.467717 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.468213 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.468357 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:09.470841 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:09.470846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:09.470859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:09.470884 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:09.473083 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.473549 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.473554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.473559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.473670 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.473956 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.473960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.473961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.474506 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:09.474716 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:09.474971 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:31:09.475286 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":136,"threads_started":1}
I20260620 14:31:09.476598 12549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:09.477291 12544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:09.478143 12550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba85fc53b42405b9683bfeb236dfe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } } }
I20260620 14:31:09.478255 12550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:09.478569 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba85fc53b42405b9683bfeb236dfe35. Latest consensus state: current_term: 1 leader_uuid: "0ba85fc53b42405b9683bfeb236dfe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } } }
I20260620 14:31:09.478641 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:09.478878 12558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:09.482438 12558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:09.488072 12558 catalog_manager.cc:1347] Generated new cluster ID: 51756c07c0714ef89abbb28ebbdca630
I20260620 14:31:09.488219 12558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:09.499673 12558 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:09.500806 12558 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:09.506394 12558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Generated new TSK 0
I20260620 14:31:09.507270 12558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:31:09.535624 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:09.646744 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:09.647012 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:09.647073 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:09.651033 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:09.651101 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:09.651181 12570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:09.656384 12570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:09.657490 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:09.658568 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:09.668131 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:09.668139 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.668246 12570 server_base.cc:1034] running on GCE node
W20260620 14:31:09.668160 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.668915 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:09.669570 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:09.670747 12570 hybrid_clock.cc:648] HybridClock initialized: now 1781965869670728 us; error 31 us; skew 500 ppm
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:09.679169 12570 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:31:09.680450 12570 webserver.cc:460] Webserver started at http://127.4.58.65:42597/ using document root <none> and password file <none>
I20260620 14:31:09.681207 12570 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:09.681293 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:09.681535 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:09.684206 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "0b4509d248db4546a4df3f62970442b5"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "47e6ce1e94fa955d06ca89b6dcefc72c"
server_key_iv: "fb8a0b6d05b2e7fea196785afe44da53"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.684926 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "0b4509d248db4546a4df3f62970442b5"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "47e6ce1e94fa955d06ca89b6dcefc72c"
server_key_iv: "fb8a0b6d05b2e7fea196785afe44da53"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.689502 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:31:09.692767 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.694535 12570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:31:09.694676 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "0b4509d248db4546a4df3f62970442b5"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "47e6ce1e94fa955d06ca89b6dcefc72c"
server_key_iv: "fb8a0b6d05b2e7fea196785afe44da53"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.694808 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:09.742040 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:09.745314 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:09.745474 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:09.746105 12570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:09.746965 12570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:09.747011 12570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.747052 12570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:09.747081 12570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.755347 12570 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:39241
I20260620 14:31:09.755388 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:39241 every 8 connection(s)
I20260620 14:31:09.756278 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:09.762610 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12570
I20260620 14:31:09.762741 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:09.763038 4329 external_mini_cluster.cc:1428] Setting key 6dcce434bed0bf772ce0a39cf6c5ed06
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:09.770705 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:09.758198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:52247 (local address 127.4.58.126:44693)
0620 14:31:09.758352 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:09.758356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:09.759470 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 14:31:09.760486 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:09.760491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:09.760557 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:09.760641 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:09.761903 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.762485 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.763769 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.764075 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.764185 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:09.766898 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:09.766901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:09.766902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:09.766931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:09.768488 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.769109 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.769113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.769114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.769203 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.769546 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.769548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.769549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.769735 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:09.769820 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:09.770384 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 14:31:09.770525 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:31:09.771560 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:09.758578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:52247)
0620 14:31:09.759292 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:09.759337 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:09.760196 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:09.760866 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:09.760873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:09.761251 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:09.761750 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:09.761765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.762617 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.762620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:09.763634 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:09.763645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.764256 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.764261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:09.764465 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:09.765095 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:09.765113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:09.766715 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:09.768613 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:09.768620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:09.768638 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:09.769005 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:09.769290 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:09.769293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:09.769295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:09.769446 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:09.769838 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:09.769859 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:09.770128 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 14:31:09.770299 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 14:31:09.770669 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":122,"threads_started":1}
I20260620 14:31:09.772994 12699 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44693
I20260620 14:31:09.773252 12699 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:09.773906 12699 heartbeater.cc:511] Master 127.4.58.126:44693 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:31:09.775730 12508 ts_manager.cc:194] Registered new tserver with Master: 0b4509d248db4546a4df3f62970442b5 (127.4.58.65:39241)
I20260620 14:31:09.776973 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:52247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:31:09.789232 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:09.898593 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:09.898873 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:09.898916 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:09.903033 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:09.903102 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:09.903164 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:09.908502 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:09.909791 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:09.910844 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:09.920795 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:09.920804 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.920931 12704 server_base.cc:1034] running on GCE node
W20260620 14:31:09.921125 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:09.921545 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:09.922191 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:09.923415 12704 hybrid_clock.cc:648] HybridClock initialized: now 1781965869923328 us; error 49 us; skew 500 ppm
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:09.926180 12704 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:31:09.927176 12704 webserver.cc:460] Webserver started at http://127.4.58.66:34777/ using document root <none> and password file <none>
I20260620 14:31:09.927735 12704 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:09.927799 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:09.927995 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:09.930085 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "9343c35c15034a5e8303d31173b1c624"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "e4693d897ec543ec93760cf2a4e2309b"
server_key_iv: "b3d4e79219c5cf5ebc162af6c433ebda"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.930640 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "9343c35c15034a5e8303d31173b1c624"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "e4693d897ec543ec93760cf2a4e2309b"
server_key_iv: "b3d4e79219c5cf5ebc162af6c433ebda"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.934475 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:09.936941 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.938264 12704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:31:09.938378 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "9343c35c15034a5e8303d31173b1c624"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "e4693d897ec543ec93760cf2a4e2309b"
server_key_iv: "b3d4e79219c5cf5ebc162af6c433ebda"
server_key_version: "encryptionkey@0"
I20260620 14:31:09.938508 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:09.955708 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:09.958931 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:09.959113 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:09.959695 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:09.960505 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:09.960551 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.960603 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:09.960632 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:09.970263 12704 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:35021
I20260620 14:31:09.970278 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:35021 every 8 connection(s)
I20260620 14:31:09.971333 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:09.975553 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12704
I20260620 14:31:09.975677 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:09.975963 4329 external_mini_cluster.cc:1428] Setting key ce4317a354ef69c6b95c26d88ec81ab1
Jun 20 14:31:09 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965869, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:31:09.986639 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:09.973665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:38217 (local address 127.4.58.126:44693)
0620 14:31:09.973814 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:09.973819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:09.974604 (+ 785us) server_negotiation.cc:407] Connection header received
0620 14:31:09.976250 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:09.976253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:09.976302 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:09.976376 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:09.977838 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.978420 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.979740 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.979968 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.980055 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:09.982701 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:09.982704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:09.982706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:09.982734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:09.984301 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.985011 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.985015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.985017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.985109 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:09.985490 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:09.985493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:09.985494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:09.985714 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:09.985796 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:09.986368 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 14:31:09.986508 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:31:09.987541 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:09.973879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:38217)
0620 14:31:09.974440 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:09.974495 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:09.975985 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:09.976581 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:09.976589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:09.977109 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:09.977701 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:09.977715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.978549 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.978553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:09.979632 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:09.979641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:09.980107 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:09.980110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:09.980274 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:09.980898 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:09.980914 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:09.982502 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:09.984452 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:09.984459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:09.984478 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:09.984876 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:09.985203 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:09.985206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:09.985208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:09.985369 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:09.985821 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:09.985827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:09.986129 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:31:09.986290 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 14:31:09.986620 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":147,"threads_started":1}
I20260620 14:31:09.988775 12833 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44693
I20260620 14:31:09.989058 12833 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:09.989557 12833 heartbeater.cc:511] Master 127.4.58.126:44693 requested a full tablet report, sending...
I20260620 14:31:09.990602 12508 ts_manager.cc:194] Registered new tserver with Master: 9343c35c15034a5e8303d31173b1c624 (127.4.58.66:35021)
I20260620 14:31:09.991331 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:38217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:31:10.001775 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:10.113531 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:10.113824 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:10.113884 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:10.118088 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:10.118157 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:10.118242 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:10.123760 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:10.124972 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:10.126091 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:10.136313 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.136488 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.136384 12838 server_base.cc:1034] running on GCE node
W20260620 14:31:10.136317 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.137092 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:10.137913 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:10.139127 12838 hybrid_clock.cc:648] HybridClock initialized: now 1781965870139097 us; error 43 us; skew 500 ppm
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:10.142184 12838 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:31:10.143216 12838 webserver.cc:460] Webserver started at http://127.4.58.67:40819/ using document root <none> and password file <none>
I20260620 14:31:10.143807 12838 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:10.143889 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:10.144104 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:10.146268 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "405622c1851d40b29327870ad9bc2579"
format_stamp: "Formatted at 2026-06-20 14:31:10 on dist-test-slave-gqzk"
server_key: "568ea956d3514543a6638aeb1809df01"
server_key_iv: "e7014283f8493feae2b871c4f9b16b68"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.146862 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "405622c1851d40b29327870ad9bc2579"
format_stamp: "Formatted at 2026-06-20 14:31:10 on dist-test-slave-gqzk"
server_key: "568ea956d3514543a6638aeb1809df01"
server_key_iv: "e7014283f8493feae2b871c4f9b16b68"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.151021 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:10.153621 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.154960 12838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:10.155076 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "405622c1851d40b29327870ad9bc2579"
format_stamp: "Formatted at 2026-06-20 14:31:10 on dist-test-slave-gqzk"
server_key: "568ea956d3514543a6638aeb1809df01"
server_key_iv: "e7014283f8493feae2b871c4f9b16b68"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.155182 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:10.169169 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:10.172351 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:10.172544 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:10.173167 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:10.173987 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:10.174033 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.174088 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:10.174115 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.182875 12838 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:45727
I20260620 14:31:10.182907 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:45727 every 8 connection(s)
I20260620 14:31:10.183784 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:10.188289 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12838
I20260620 14:31:10.188409 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:10.188666 4329 external_mini_cluster.cc:1428] Setting key 7ca4837cf97b6f698c49a0c13223f52b
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:10.197854 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.185604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:50279 (local address 127.4.58.126:44693)
0620 14:31:10.185733 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.185737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.186691 (+ 954us) server_negotiation.cc:407] Connection header received
0620 14:31:10.187978 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.187982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.188074 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.188146 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.189619 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.190308 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.191650 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.191866 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.191950 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.194461 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:10.194464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:10.194466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:10.194491 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:10.195829 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:10.196322 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:10.196326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:10.196327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:10.196401 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:10.196720 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:10.196723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:10.196725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:10.196941 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:10.197028 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:10.197565 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 14:31:10.197689 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:31:10.198683 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.185998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:50279)
0620 14:31:10.186520 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.186567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.187762 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.188338 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.188347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.188816 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.189448 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.189464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.190442 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.190446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.191511 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.191521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.192012 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.192016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.192209 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.192785 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.192808 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.194289 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:10.195941 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:10.195947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:10.195964 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:10.196229 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:10.196483 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:10.196486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:10.196489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:10.196635 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:10.197035 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:10.197042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:10.197329 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 14:31:10.197503 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:31:10.197841 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":122,"threads_started":1}
I20260620 14:31:10.199946 12967 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44693
I20260620 14:31:10.200220 12967 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:10.200815 12967 heartbeater.cc:511] Master 127.4.58.126:44693 requested a full tablet report, sending...
I20260620 14:31:10.201898 12508 ts_manager.cc:194] Registered new tserver with Master: 405622c1851d40b29327870ad9bc2579 (127.4.58.67:45727)
I20260620 14:31:10.202476 12508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:50279
I20260620 14:31:10.202837 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
Entry for principal oryx/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab.
WARNING: no policy specified for oryx/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
Entry for principal oryx/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab.
WARNING: no policy specified for oryx/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
Entry for principal oryx/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab.
I20260620 14:31:10.276183 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12570
I20260620 14:31:10.281180 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12704
I20260620 14:31:10.285681 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12838
I20260620 14:31:10.290679 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12476
I20260620 14:31:10.296614 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44693
--webserver_interface=127.4.58.126
--webserver_port=43511
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:10.407652 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:10.407927 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:10.407967 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:10.411989 12976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:10.412061 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:10.412079 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:10.412094 12976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:10.412108 12976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:10.417512 12976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=43511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:10.418701 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:10.419730 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:10.428009 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.428005 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.428009 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.428077 12976 server_base.cc:1034] running on GCE node
I20260620 14:31:10.428846 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:10.429965 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:10.431149 12976 hybrid_clock.cc:648] HybridClock initialized: now 1781965870431127 us; error 31 us; skew 500 ppm
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965870, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:10.433898 12976 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:31:10.434772 12976 webserver.cc:460] Webserver started at http://127.4.58.126:43511/ using document root <none> and password file <none>
I20260620 14:31:10.435293 12976 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:10.435372 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:10.439651 12976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:31:10.441902 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.443285 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:10.443423 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "0ba85fc53b42405b9683bfeb236dfe35"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "4b11ef64ab394f5d1a6630be6d8de68a"
server_key_iv: "28ebf78b28f5239407ad1f8afb122b09"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.443986 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:10.458726 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:10.461822 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:10.462109 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:10.468835 12976 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44693
I20260620 14:31:10.468868 13043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44693 every 8 connection(s)
I20260620 14:31:10.469897 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:10.472851 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 12976
I20260620 14:31:10.473255 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:39241
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:44693
--webserver_port=42597
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:31:10.474278 13044 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:31:10.476271 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Bootstrap starting.
I20260620 14:31:10.492498 13044 log.cc:824] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:10.499281 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:31:10.499637 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Bootstrap complete.
I20260620 14:31:10.503656 13044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:10.504038 13044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba85fc53b42405b9683bfeb236dfe35, State: Initialized, Role: FOLLOWER
I20260620 14:31:10.504530 13044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:10.504644 13044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:10.504705 13044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:10.504819 13044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:31:10.506618 13044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:10.506909 13044 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba85fc53b42405b9683bfeb236dfe35; no voters:
I20260620 14:31:10.507203 13044 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:31:10.507306 13048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:31:10.507484 13048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [term 2 LEADER]: Becoming Leader. State: Replica: 0ba85fc53b42405b9683bfeb236dfe35, State: Running, Role: LEADER
I20260620 14:31:10.507777 13048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } }
I20260620 14:31:10.508387 13044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:10.508708 13048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba85fc53b42405b9683bfeb236dfe35. Latest consensus state: current_term: 2 leader_uuid: "0ba85fc53b42405b9683bfeb236dfe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } } }
I20260620 14:31:10.508920 13048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:10.510516 13049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0ba85fc53b42405b9683bfeb236dfe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba85fc53b42405b9683bfeb236dfe35" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44693 } } }
I20260620 14:31:10.510638 13049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:10.510757 13052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:10.513317 13052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:10.513840 13052 catalog_manager.cc:1259] Loaded cluster ID: 51756c07c0714ef89abbb28ebbdca630
I20260620 14:31:10.513890 13052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:10.515502 13052 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:10.516043 13052 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0ba85fc53b42405b9683bfeb236dfe35: Loaded TSK: 0
I20260620 14:31:10.517040 13052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:31:10.613106 13046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:10.613482 13046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:10.613556 13046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:10.619493 13046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:10.619589 13046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:10.619675 13046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:10.627517 13046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.65.keytab
--principal=oryx/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:39241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=42597
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:10.628913 13046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:10.630205 13046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:10.641027 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.641027 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.641093 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.641906 13046 server_base.cc:1034] running on GCE node
I20260620 14:31:10.642406 13046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:10.643208 13046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:10.644394 13046 hybrid_clock.cc:648] HybridClock initialized: now 1781965870644363 us; error 44 us; skew 500 ppm
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:10.647891 13046 init.cc:377] Logged in from keytab as oryx/127.4.58.65@KRBTEST.COM (short username oryx)
I20260620 14:31:10.649065 13046 webserver.cc:460] Webserver started at http://127.4.58.65:42597/ using document root <none> and password file <none>
I20260620 14:31:10.649782 13046 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:10.649887 13046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:10.655061 13046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:31:10.657775 13080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.659405 13046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:10.659552 13046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "0b4509d248db4546a4df3f62970442b5"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "47e6ce1e94fa955d06ca89b6dcefc72c"
server_key_iv: "fb8a0b6d05b2e7fea196785afe44da53"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.660295 13046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:10.698844 13046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:10.702649 13046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:10.702841 13046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:10.703518 13046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:10.704500 13046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:10.704574 13046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.704627 13046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:10.704654 13046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.713721 13046 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:39241
I20260620 14:31:10.713739 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:39241 every 8 connection(s)
I20260620 14:31:10.714744 13046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:10.720808 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13046
I20260620 14:31:10.721256 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:35021
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:44693
--webserver_port=34777
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:10.728224 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.716986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:49871)
0620 14:31:10.717675 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.717715 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.718518 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.719617 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.719624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.719968 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.720451 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.720467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.722016 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.722020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.723138 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.723149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.724154 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.724157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.724301 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.724925 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.724942 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.727427 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":147,"threads_started":1}
W20260620 14:31:10.728780 13194 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:44693 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.58.126:44693: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
W20260620 14:31:10.729282 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.716818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:49871 (local address 127.4.58.126:44693)
0620 14:31:10.717257 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.717283 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.717826 (+ 543us) server_negotiation.cc:407] Connection header received
0620 14:31:10.718905 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.718918 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.719228 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.719503 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.720643 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.721857 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.723265 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.723991 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.724157 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.727728 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:49871: BlockingRecv error: recv got EOF from 127.4.58.65:49871 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:10.734917 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.729221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:37541)
0620 14:31:10.729564 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.729577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.729645 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.730087 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.730089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.730215 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.730305 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.730310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.731408 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.731411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.731953 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.731961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.732523 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.732526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.732596 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.733073 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.733083 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.734817 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 14:31:10.735098 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.729303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:37541 (local address 127.4.58.126:44693)
0620 14:31:10.729867 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.729871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.729888 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:10.729931 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.729934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.729985 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.730074 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.730460 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.731234 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.732059 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.732412 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.732510 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.734980 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:37541: BlockingRecv error: recv got EOF from 127.4.58.65:37541 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":132,"threads_started":1}
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:10.740146 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.735492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:55419)
0620 14:31:10.735604 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.735618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.735695 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.735921 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.735924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.736052 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.736119 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.736123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.736889 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.736892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.737419 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.737427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.737817 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.737820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.737901 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.738346 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.738357 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.740026 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:31:10.740271 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.735568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55419 (local address 127.4.58.126:44693)
0620 14:31:10.735668 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.735672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.735685 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:31:10.735777 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.735780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.735821 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.735883 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.736215 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.736790 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.737519 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.737728 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.737809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.740170 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:55419: BlockingRecv error: recv got EOF from 127.4.58.65:55419 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:31:10.740535 13194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:31:10.834302 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:10.834595 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:10.834659 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:10.839030 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:10.839105 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:10.839188 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:10.844570 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.66.keytab
--principal=oryx/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:35021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=34777
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:10.845711 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:10.846772 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:10.856662 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:10.856686 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.856879 13198 server_base.cc:1034] running on GCE node
W20260620 14:31:10.856686 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:10.857342 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:10.857996 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:10.859187 13198 hybrid_clock.cc:648] HybridClock initialized: now 1781965870859152 us; error 44 us; skew 500 ppm
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965870, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:10.862152 13198 init.cc:377] Logged in from keytab as oryx/127.4.58.66@KRBTEST.COM (short username oryx)
I20260620 14:31:10.863082 13198 webserver.cc:460] Webserver started at http://127.4.58.66:34777/ using document root <none> and password file <none>
I20260620 14:31:10.863633 13198 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:10.863715 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:10.867774 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:31:10.870064 13214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.871529 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:31:10.871659 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "9343c35c15034a5e8303d31173b1c624"
format_stamp: "Formatted at 2026-06-20 14:31:09 on dist-test-slave-gqzk"
server_key: "e4693d897ec543ec93760cf2a4e2309b"
server_key_iv: "b3d4e79219c5cf5ebc162af6c433ebda"
server_key_version: "encryptionkey@0"
I20260620 14:31:10.872247 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:10.898562 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:10.902179 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:10.902375 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:10.903012 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:10.903883 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:10.903934 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.903986 13198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:10.904016 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:10.912992 13198 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:35021
I20260620 14:31:10.913028 13327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:35021 every 8 connection(s)
I20260620 14:31:10.914047 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:10.918507 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13198
I20260620 14:31:10.919031 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:45727
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:44693
--webserver_port=40819
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:10.926517 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.916013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:39587 (local address 127.4.58.126:44693)
0620 14:31:10.916170 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.916174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.917261 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 14:31:10.918202 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.918205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.918257 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.918335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.919974 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.920527 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.922212 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.922506 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.922605 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.926409 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:39587: BlockingRecv error: recv got EOF from 127.4.58.66:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:31:10.927366 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.916401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:39587)
0620 14:31:10.917114 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.917161 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.917952 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.918547 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.918557 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.919119 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.919796 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.919814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.920672 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.920675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.922065 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.922077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.922672 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.922676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.922878 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.923489 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.923512 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.926253 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":111,"threads_started":1}
W20260620 14:31:10.927879 13328 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:44693 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.58.126:44693: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:10.933192 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.928306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:59943)
0620 14:31:10.928600 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.928614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.928703 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.928941 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.928944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.929101 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.929193 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.929198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.929885 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.929889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.930429 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.930436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.930843 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.930847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.930930 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.931366 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.931379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.933060 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232}
W20260620 14:31:10.933348 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.928429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:59943 (local address 127.4.58.126:44693)
0620 14:31:10.928599 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.928603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.928657 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:31:10.928807 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.928809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.928851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.928916 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.929297 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.929790 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.930542 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.930751 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.930833 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.933204 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:59943: BlockingRecv error: recv got EOF from 127.4.58.66:59943 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:10 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:10.938967 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:10.933803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:60129)
0620 14:31:10.933953 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:10.933965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:10.934033 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:10.934240 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:10.934243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:10.934399 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:10.934485 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.934491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.935212 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.935215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.936085 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:10.936095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.936508 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.936511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:10.936598 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.937079 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:10.937104 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:10.938835 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 14:31:10.939093 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:10.933882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:60129 (local address 127.4.58.126:44693)
0620 14:31:10.933986 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:10.933989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:10.934000 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:10.934110 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:10.934112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:10.934147 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:10.934213 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:10.934622 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.935108 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.936200 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:10.936403 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:10.936537 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:10.938978 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:60129: BlockingRecv error: recv got EOF from 127.4.58.66:60129 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 14:31:10.939317 13328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:31:11.032491 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:11.032804 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:11.032869 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:11.036967 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:11.037043 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:11.037140 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:11.042675 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/krb5kdc/oryx_127.4.58.67.keytab
--principal=oryx/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:45727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=40819
--tserver_master_addrs=127.4.58.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:11.043797 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:11.044924 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:11.054867 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:11.054942 13331 server_base.cc:1034] running on GCE node
W20260620 14:31:11.054839 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:11.054839 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:11.055579 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:11.056227 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:11.057380 13331 hybrid_clock.cc:648] HybridClock initialized: now 1781965871057344 us; error 48 us; skew 500 ppm
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965871, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:11.060103 13331 init.cc:377] Logged in from keytab as oryx/127.4.58.67@KRBTEST.COM (short username oryx)
I20260620 14:31:11.061015 13331 webserver.cc:460] Webserver started at http://127.4.58.67:40819/ using document root <none> and password file <none>
I20260620 14:31:11.061537 13331 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:11.061674 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:11.065630 13331 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:31:11.067749 13346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:11.069037 13331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:11.069146 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "405622c1851d40b29327870ad9bc2579"
format_stamp: "Formatted at 2026-06-20 14:31:10 on dist-test-slave-gqzk"
server_key: "568ea956d3514543a6638aeb1809df01"
server_key_iv: "e7014283f8493feae2b871c4f9b16b68"
server_key_version: "encryptionkey@0"
I20260620 14:31:11.069679 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:11.092554 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:11.095728 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:11.095907 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:11.096515 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:11.097391 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:11.097440 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:11.097493 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:11.097523 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:11.106261 13331 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:45727
I20260620 14:31:11.106304 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:45727 every 8 connection(s)
I20260620 14:31:11.107303 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:11.116072 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13331
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:31:11.118767 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:11.109227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:40343 (local address 127.4.58.126:44693)
0620 14:31:11.109424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.109428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.110636 (+ 1208us) server_negotiation.cc:407] Connection header received
0620 14:31:11.111545 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.111548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.111603 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.111682 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.112949 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.113510 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.114650 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.114864 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.114947 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.118639 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:40343: BlockingRecv error: recv got EOF from 127.4.58.67:40343 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:31:11.119385 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.109737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:40343)
0620 14:31:11.110416 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:11.110539 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:11.111318 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:11.111849 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:11.111856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:11.112217 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:11.112810 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.112826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.113633 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.113636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.114532 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.114541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.114987 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.114990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.115128 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.115674 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:11.115690 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:11.118504 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":120,"threads_started":1}
W20260620 14:31:11.119870 13460 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:44693 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.58.126:44693: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:11.126318 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.117104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.4.58.126:44693)
0620 14:31:11.117240 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.117252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.117263 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:31:11.117297 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.117299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.117341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.117411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.118297 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.119015 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.119995 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.120218 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.120302 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.121874 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:11.121880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:11.121895 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:11.121922 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:11.123898 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:11.124406 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:11.124412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:11.124417 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:11.124598 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:11.124904 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:11.124908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:11.124910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:11.125374 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:11.125631 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:11.125911 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:31:11.126164 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:31:11.126492 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.120784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:60943)
0620 14:31:11.120894 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:11.120908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:11.120983 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:11.121215 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:11.121218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:11.121366 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:11.121493 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.121501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.122258 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.122262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.122943 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.122953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.123528 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.123531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.123600 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.123974 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:11.123989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:11.126381 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:31:11.126685 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:11.120843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:60943 (local address 127.4.58.126:44693)
0620 14:31:11.120942 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.120945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.120959 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:11.121065 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.121068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.121105 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.121165 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.121639 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.122162 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.123066 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.123384 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.123498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.126587 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:60943: BlockingRecv error: recv got EOF from 127.4.58.67:60943 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:11.132076 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.127207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:57289)
0620 14:31:11.127300 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:11.127318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:11.127377 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:11.127780 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:11.127783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:11.127915 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:11.128000 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.128005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.128787 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.128791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.129347 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.129355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.129745 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.129748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.129822 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.130271 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:11.130282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:11.131948 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:31:11.132164 13196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:11.127468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:57289 (local address 127.4.58.126:44693)
0620 14:31:11.127598 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.127602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.127611 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:31:11.127642 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.127644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.127677 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.127739 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.128096 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.128655 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.129454 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.129655 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.129737 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.132065 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:57289: BlockingRecv error: recv got EOF from 127.4.58.67:57289 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 14:31:11.132421 13460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:11.746752 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.741218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:40357)
0620 14:31:11.741569 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:11.741586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:11.741685 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:11.741977 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:11.741980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:11.742154 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:11.742254 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.742260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.743088 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.743091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.743707 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.743717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.744203 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.744206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.744282 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.744811 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:11.744827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:11.746606 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":95,"threads_started":1}
W20260620 14:31:11.746933 13465 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:11.741322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:40357 (local address 127.4.58.126:44693)
0620 14:31:11.741565 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.741569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.741641 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:31:11.741755 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.741757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.741812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.741907 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.742405 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.742965 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.743840 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.744074 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.744164 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.746787 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:40357: BlockingRecv error: recv got EOF from 127.4.58.65:40357 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:11 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:11.945773 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:11.940060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:51073)
0620 14:31:11.940362 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:11.940377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:11.940555 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:11.940861 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:11.940864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:11.941061 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:11.941156 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.941162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.941904 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.941908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.942501 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:11.942511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.943027 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.943031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:11.943123 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.943700 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:11.943712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:11.945610 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
W20260620 14:31:11.945854 13465 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:11.940214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:51073 (local address 127.4.58.126:44693)
0620 14:31:11.940364 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:11.940368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:11.940455 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:31:11.940660 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:11.940663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:11.940725 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:11.940816 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:11.941276 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.941799 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.942621 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:11.942863 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:11.942977 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:11.945755 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:51073: BlockingRecv error: recv got EOF from 127.4.58.66:51073 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:12.139000 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:12.133224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:41207)
0620 14:31:12.133629 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:12.133642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:12.133774 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:12.134037 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:12.134040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:12.134235 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:12.134328 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.134334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.135187 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.135191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.135792 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.135802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.136264 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.136268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.136353 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.136921 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:12.136934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:12.138834 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":89,"threads_started":1}
W20260620 14:31:12.139096 13465 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:12.133373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:41207 (local address 127.4.58.126:44693)
0620 14:31:12.133552 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:12.133557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:12.133721 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:31:12.133868 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:12.133871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:12.133924 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:12.134002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:12.134446 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.135097 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.135910 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.136144 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.136232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.138994 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:41207: BlockingRecv error: recv got EOF from 127.4.58.67:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:12.753575 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:12.747705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:40801)
0620 14:31:12.748016 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:12.748032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:12.748160 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:12.748487 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:12.748492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:12.748733 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:12.748868 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.748874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.749668 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.749672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.750238 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.750246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.750705 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.750708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.750793 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.751343 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:12.751356 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:12.753385 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
W20260620 14:31:12.753702 13469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:12.747782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:40801 (local address 127.4.58.126:44693)
0620 14:31:12.748074 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:12.748078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:12.748096 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:31:12.748250 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:12.748253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:12.748308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:12.748401 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:12.749009 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.749555 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.750355 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.750599 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.750688 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.753580 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:40801: BlockingRecv error: recv got EOF from 127.4.58.65:40801 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:12 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:12.952386 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:12.946713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:34389)
0620 14:31:12.947037 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:12.947053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:12.947199 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:12.947470 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:12.947473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:12.947687 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:12.947783 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.947789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.948530 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.948534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.949170 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:12.949180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.949614 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.949618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:12.949700 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.950200 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:12.950213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:12.952241 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
W20260620 14:31:12.952510 13469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:12.946869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34389 (local address 127.4.58.126:44693)
0620 14:31:12.947040 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:12.947044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:12.947158 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:31:12.947306 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:12.947309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:12.947358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:12.947428 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:12.947903 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.948422 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.949283 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:12.949508 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:12.949595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:12.952388 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:34389: BlockingRecv error: recv got EOF from 127.4.58.66:34389 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:13.145778 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:13.139921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:38305)
0620 14:31:13.140248 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:13.140269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:13.140374 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:13.140634 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:13.140636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:13.140844 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:13.140955 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.140961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.141764 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.141767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.142364 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.142373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.142847 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.142851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.142922 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.143449 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:13.143464 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:13.145630 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":86,"threads_started":1}
W20260620 14:31:13.145889 13469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:13.140178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:38305 (local address 127.4.58.126:44693)
0620 14:31:13.140362 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:13.140365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:13.140382 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:13.140476 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:13.140478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:13.140531 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:13.140594 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:13.141065 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.141639 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.142509 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.142724 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.142801 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.145771 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:38305: BlockingRecv error: recv got EOF from 127.4.58.67:38305 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:13.760250 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:13.754500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:45881)
0620 14:31:13.754799 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:13.754817 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:13.754967 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:13.755214 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:13.755216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:13.755393 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:13.755496 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.755502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.756299 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.756302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.756994 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.757007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.757541 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.757546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.757644 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.758135 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:13.758147 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:13.760106 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20260620 14:31:13.760535 13473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:13.754683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:45881 (local address 127.4.58.126:44693)
0620 14:31:13.754971 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:13.754975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:13.754991 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:31:13.755039 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:13.755041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:13.755082 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:13.755169 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:13.755611 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.756165 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.757121 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.757389 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.757499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.760323 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:45881: BlockingRecv error: recv got EOF from 127.4.58.65:45881 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:13 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:13.958663 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:13.953178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:44945)
0620 14:31:13.953437 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:13.953451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:13.953612 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:13.953862 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:13.953864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:13.954054 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:13.954156 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.954161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.954917 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.954920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.955507 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:13.955516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.955970 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.955973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:13.956063 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.956593 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:13.956606 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:13.958523 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20260620 14:31:13.958843 13473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:13.953294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:44945 (local address 127.4.58.126:44693)
0620 14:31:13.953499 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:13.953503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:13.953639 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:31:13.953688 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:13.953690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:13.953738 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:13.953822 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:13.954254 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.954799 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.955615 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:13.955834 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:13.955928 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:13.958711 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:44945: BlockingRecv error: recv got EOF from 127.4.58.66:44945 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:14.152251 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:14.146637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:38151)
0620 14:31:14.146946 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:14.146960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:14.147064 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:14.147303 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:14.147305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:14.147537 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:14.147691 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.147700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.148517 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.148520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.149179 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.149189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.149702 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.149705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.149774 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.150276 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:14.150287 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:14.152103 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
W20260620 14:31:14.152451 13473 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:14.146766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:38151 (local address 127.4.58.126:44693)
0620 14:31:14.146916 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:14.146921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:14.147028 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:31:14.147145 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:14.147147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:14.147195 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:14.147263 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:14.147847 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.148386 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.149328 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.149572 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.149664 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.152283 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:38151: BlockingRecv error: recv got EOF from 127.4.58.67:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:14.771571 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:14.761033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:55493)
0620 14:31:14.761325 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:14.761338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:14.761428 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:14.761780 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:14.761783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:14.761981 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:14.762089 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.762095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.762929 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.762933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.763552 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.763562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.764044 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.764048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.764118 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.769362 (+ 5244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:14.769377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:14.771438 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260620 14:31:14.771806 13477 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:14.761124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55493 (local address 127.4.58.126:44693)
0620 14:31:14.761388 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:14.761391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:14.761405 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:14.761556 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:14.761558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:14.761604 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:14.761688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:14.762227 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.762799 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.763687 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.763914 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.764016 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.771667 (+ 7651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:55493: BlockingRecv error: recv got EOF from 127.4.58.65:55493 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:14 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:14.965234 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:14.959561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:46113)
0620 14:31:14.959839 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:14.959853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:14.959958 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:14.960223 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:14.960225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:14.960403 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:14.960499 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.960504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.961442 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.961447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.962254 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:14.962265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.962703 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.962706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:14.962769 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.963292 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:14.963308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:14.965085 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
W20260620 14:31:14.965305 13477 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:14.959691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:46113 (local address 127.4.58.126:44693)
0620 14:31:14.959841 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:14.959845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:14.959947 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:31:14.960034 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:14.960036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:14.960096 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:14.960183 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:14.960607 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.961311 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.962357 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:14.962601 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:14.962681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:14.965213 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:46113: BlockingRecv error: recv got EOF from 127.4.58.66:46113 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:15.158385 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:15.153257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:35471)
0620 14:31:15.153536 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:15.153549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:15.153660 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:15.153908 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:15.153910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:15.154105 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:15.154196 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.154202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.154986 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.154990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.155530 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.155537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.155950 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.155954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.156036 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.156485 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:15.156495 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:15.158236 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
W20260620 14:31:15.158485 13477 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:15.153343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:35471 (local address 127.4.58.126:44693)
0620 14:31:15.153484 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:15.153496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:15.153628 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:31:15.153751 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:15.153754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:15.153805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:15.153869 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:15.154305 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.154870 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.155634 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.155844 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.155928 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.158376 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:35471: BlockingRecv error: recv got EOF from 127.4.58.67:35471 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:15.778009 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:15.772511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:56633)
0620 14:31:15.772822 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:15.772835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:15.772937 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:15.773203 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:15.773205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:15.773383 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:15.773479 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.773484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.774269 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.774272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.774878 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.774888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.775433 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.775437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.775510 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.775963 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:15.775999 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:15.777879 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
W20260620 14:31:15.778200 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:15.772510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:56633 (local address 127.4.58.126:44693)
0620 14:31:15.772791 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:15.772795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:15.772894 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:31:15.773023 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:15.773025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:15.773092 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:15.773169 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:15.773578 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.774145 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.775044 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.775274 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.775378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.778063 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:56633: BlockingRecv error: recv got EOF from 127.4.58.65:56633 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:15 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:15.972810 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:15.965991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:49423)
0620 14:31:15.966289 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:15.966306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:15.966425 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:15.966764 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:15.966768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:15.966994 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:15.967115 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.967123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.968129 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.968132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.969119 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:15.969132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.969689 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.969693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:15.969781 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.970359 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:15.970373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:15.972621 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20260620 14:31:15.972968 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:15.966189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:49423 (local address 127.4.58.126:44693)
0620 14:31:15.966327 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:15.966331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:15.966370 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:31:15.966602 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:15.966605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:15.966661 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:15.966741 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:15.967229 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.968013 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.969259 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:15.969580 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:15.969676 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:15.972849 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:49423: BlockingRecv error: recv got EOF from 127.4.58.66:49423 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:16.164853 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:16.159264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:47541)
0620 14:31:16.159527 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:16.159549 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:16.159643 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:16.159920 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:16.159922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:16.160135 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:16.160231 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.160236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.161046 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.161048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.161648 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.161658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.162145 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.162148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.162229 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.162739 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:16.162751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:16.164687 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260620 14:31:16.165243 13480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:16.159361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:47541 (local address 127.4.58.126:44693)
0620 14:31:16.159521 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:16.159524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:16.159613 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:31:16.159719 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:16.159721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:16.159765 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:16.159843 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:16.160367 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.160938 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.161770 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.162003 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.162094 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.164893 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:47541: BlockingRecv error: recv got EOF from 127.4.58.67:47541 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:16.783967 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:16.778778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:33769)
0620 14:31:16.779036 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:16.779051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:16.779137 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:16.779393 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:16.779396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:16.779602 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:16.779705 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.779711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.780412 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.780414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.781061 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.781070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.781541 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.781544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.781611 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.782140 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:16.782152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:16.783848 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
W20260620 14:31:16.784147 13485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:16.778857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:33769 (local address 127.4.58.126:44693)
0620 14:31:16.779096 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:16.779099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:16.779111 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:16.779218 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:16.779220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:16.779264 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:16.779338 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:16.779800 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.780320 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.781175 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.781401 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.781521 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.784006 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:33769: BlockingRecv error: recv got EOF from 127.4.58.65:33769 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:16 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:16.978929 13486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:16.973549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:58891)
0620 14:31:16.973817 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:16.973830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:16.973929 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:16.974181 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:16.974183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:16.974442 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:16.974541 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.974547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.975328 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.975333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.975979 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:16.975989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.976514 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.976518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:16.976592 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.977085 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:16.977096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:16.978812 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
W20260620 14:31:16.979095 13485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:16.973615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:58891 (local address 127.4.58.126:44693)
0620 14:31:16.973763 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:16.973767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:16.973892 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:31:16.974036 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:16.974038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:16.974085 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:16.974155 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:16.974641 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.975178 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.976102 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:16.976322 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:16.976428 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:16.978979 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:58891: BlockingRecv error: recv got EOF from 127.4.58.66:58891 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:17.170912 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:17.165675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:46925)
0620 14:31:17.165956 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:17.165971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:17.166101 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:17.166355 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:17.166358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:17.166551 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:17.166653 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.166658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.167485 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.167487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.168061 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.168070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.168465 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.168468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.168530 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.169070 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:17.169082 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:17.170791 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260620 14:31:17.171221 13485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:17.165796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46925 (local address 127.4.58.126:44693)
0620 14:31:17.165985 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:17.165989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:17.166094 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:31:17.166197 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:17.166200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:17.166255 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:17.166323 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:17.166802 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.167381 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.168162 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.168377 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.168454 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.171000 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:46925: BlockingRecv error: recv got EOF from 127.4.58.67:46925 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:17.790153 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:17.784712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:43987)
0620 14:31:17.785095 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:17.785109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:17.785213 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:17.785478 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:17.785481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:17.785661 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:17.785763 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.785768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.786604 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.786606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.787171 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.787179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.787649 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.787653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.787738 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.788303 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:17.788315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:17.790033 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":69,"threads_started":1}
W20260620 14:31:17.790325 13489 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:17.784834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:43987 (local address 127.4.58.126:44693)
0620 14:31:17.785089 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:17.785093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:17.785176 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:31:17.785284 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:17.785286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:17.785329 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:17.785406 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:17.785908 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.786461 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.787273 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.787519 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.787598 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.790184 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:43987: BlockingRecv error: recv got EOF from 127.4.58.65:43987 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:17 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:17.985040 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:17.979732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:33469)
0620 14:31:17.980061 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:17.980073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:17.980186 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:17.980424 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:17.980428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:17.980632 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:17.980726 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.980732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.981497 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.981500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.982183 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:17.982194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.982715 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.982720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:17.982784 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.983219 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:17.983229 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:17.984923 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
W20260620 14:31:17.985210 13489 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:17.979826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:33469 (local address 127.4.58.126:44693)
0620 14:31:17.979973 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:17.979978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:17.980142 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:31:17.980271 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:17.980274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:17.980322 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:17.980388 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:17.980859 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.981391 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.982303 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:17.982604 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:17.982703 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:17.985085 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:33469: BlockingRecv error: recv got EOF from 127.4.58.66:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:18.177438 13491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:18.171742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:55423)
0620 14:31:18.172091 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:18.172105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:18.172225 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:18.172617 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:18.172624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:18.172861 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:18.172982 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.172990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.173836 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.173838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.174470 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.174479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.174913 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.174916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.174989 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.175528 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:18.175540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:18.177323 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
W20260620 14:31:18.177558 13489 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:18.171877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:55423 (local address 127.4.58.126:44693)
0620 14:31:18.172022 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:18.172027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:18.172186 (+ 159us) server_negotiation.cc:407] Connection header received
0620 14:31:18.172317 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:18.172320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:18.172374 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:18.172453 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:18.173099 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.173741 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.174573 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.174791 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.174868 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.177455 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:55423: BlockingRecv error: recv got EOF from 127.4.58.67:55423 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:18.796837 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:18.791046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:57571)
0620 14:31:18.791337 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:18.791352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:18.791464 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:18.791766 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:18.791769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:18.791951 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:18.792055 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.792060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.792891 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.792895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.793725 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.793739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.794317 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.794320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.794395 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.794881 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:18.794893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:18.796680 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20260620 14:31:18.796953 13493 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:18.791267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:57571 (local address 127.4.58.126:44693)
0620 14:31:18.791493 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:18.791496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:18.791510 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:18.791552 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:18.791555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:18.791603 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:18.791689 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:18.792173 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.792781 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.793865 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.794191 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.794273 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.796839 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:57571: BlockingRecv error: recv got EOF from 127.4.58.65:57571 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:18 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:18.991552 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:18.985799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:53535)
0620 14:31:18.986158 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:18.986173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:18.986277 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:18.986520 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:18.986523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:18.986723 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:18.986816 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.986821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.987700 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.987703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.988272 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:18.988280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.988725 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.988729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:18.988858 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.989418 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:18.989431 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:18.991395 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":89,"threads_started":1}
W20260620 14:31:18.991724 13493 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:18.985851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:53535 (local address 127.4.58.126:44693)
0620 14:31:18.985990 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:18.985994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:18.986247 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:31:18.986366 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:18.986369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:18.986414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:18.986488 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:18.986929 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.987589 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.988388 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:18.988618 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:18.988706 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:18.991619 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:53535: BlockingRecv error: recv got EOF from 127.4.58.66:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:19.183643 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:19.178359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:33649)
0620 14:31:19.178690 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:19.178703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:19.178818 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:19.179105 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:19.179107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:19.179285 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:19.179378 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.179383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.180116 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.180119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.180704 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.180713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.181180 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.181183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.181245 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.181712 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:19.181723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:19.183480 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":94,"threads_started":1}
W20260620 14:31:19.183727 13493 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:19.178419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:33649 (local address 127.4.58.126:44693)
0620 14:31:19.178537 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:19.178541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:19.178779 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:31:19.178909 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:19.178911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:19.178968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:19.179074 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:19.179475 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.179989 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.180855 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.181093 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.181170 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.183625 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:33649: BlockingRecv error: recv got EOF from 127.4.58.67:33649 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:19.806147 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:19.797785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:34351)
0620 14:31:19.798065 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:19.798082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:19.798181 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:19.798606 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:19.798608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:19.798788 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:19.798886 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.798892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.799729 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.799732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.800303 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.800311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.800965 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.800968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.801047 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.804160 (+ 3113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:19.804174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:19.806012 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
W20260620 14:31:19.806301 13497 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:19.797889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:34351 (local address 127.4.58.126:44693)
0620 14:31:19.798138 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:19.798141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:19.798158 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:19.798345 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:19.798349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:19.798461 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:19.798569 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:19.799006 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.799613 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.800465 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.800702 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.800810 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.806174 (+ 5364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:34351: BlockingRecv error: recv got EOF from 127.4.58.65:34351 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:19 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:19.997956 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:19.992409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:41225)
0620 14:31:19.992845 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:19.992857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:19.992964 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:19.993212 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:19.993216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:19.993411 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:19.993505 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.993510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.994228 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.994231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.994787 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:19.994794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.995208 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.995211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:19.995292 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.995778 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:19.995789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:19.997805 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":142,"threads_started":1}
W20260620 14:31:19.998055 13497 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:19.992510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:41225 (local address 127.4.58.126:44693)
0620 14:31:19.992687 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:19.992692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:19.992921 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:31:19.993044 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:19.993047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:19.993105 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:19.993170 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:19.993610 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.994126 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.994892 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:19.995110 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:19.995194 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:19.997951 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:41225: BlockingRecv error: recv got EOF from 127.4.58.66:41225 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:20.190076 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:20.184443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:55437)
0620 14:31:20.184729 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:20.184742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:20.184859 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:20.185136 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:20.185139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:20.185323 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:20.185422 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:20.185428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.186237 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.186240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:20.186824 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:20.186834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.187295 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.187298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:20.187367 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:20.187875 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:20.187887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:20.189953 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
W20260620 14:31:20.190249 13497 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:20.184546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:55437 (local address 127.4.58.126:44693)
0620 14:31:20.184671 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:20.184675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:20.184814 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:31:20.184944 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:20.184946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:20.184990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:20.185058 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:20.185575 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.186106 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.186947 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.187178 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.187260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:20.190133 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:55437: BlockingRecv error: recv got EOF from 127.4.58.67:55437 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:20.812824 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:20.806966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:48851)
0620 14:31:20.807226 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:20.807239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:20.807324 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:20.807574 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:20.807576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:20.807746 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:20.807840 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:20.807845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.808720 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.808724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:20.809461 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:20.809473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.809947 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.809950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:20.810037 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:20.810644 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:20.810656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:20.812576 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260620 14:31:20.812949 13501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:20.807081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:48851 (local address 127.4.58.126:44693)
0620 14:31:20.807315 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:20.807319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:20.807331 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:20.807415 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:20.807418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:20.807473 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:20.807549 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:20.807954 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.808585 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.809609 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:20.809840 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:20.809923 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:20.812740 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:48851: BlockingRecv error: recv got EOF from 127.4.58.65:48851 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:20 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:21.004482 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:20.998824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:36195)
0620 14:31:20.999112 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:20.999127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:20.999242 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:20.999486 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:20.999489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:20.999701 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:20.999809 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:20.999815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.000570 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.000573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.001239 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.001250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.001684 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.001688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.001779 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.002387 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.002403 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.004355 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
W20260620 14:31:21.004630 13501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:20.998947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:36195 (local address 127.4.58.126:44693)
0620 14:31:20.999087 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:20.999092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:20.999194 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:31:20.999328 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:20.999331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:20.999377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:20.999443 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:20.999924 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.000447 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.001357 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.001579 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.001666 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.004485 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:36195: BlockingRecv error: recv got EOF from 127.4.58.66:36195 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:31:21 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:21 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:21.196461 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.190931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:58813)
0620 14:31:21.191242 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:21.191258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:21.191368 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:21.191683 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:21.191686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:21.191885 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:21.191975 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.191982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.192741 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.192744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.193415 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.193425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.193859 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.193862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.193940 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.194391 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.194402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.196277 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
W20260620 14:31:21.196666 13501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:21.191060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:58813 (local address 127.4.58.126:44693)
0620 14:31:21.191190 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.191195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.191329 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:31:21.191463 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.191466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.191573 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.191660 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.192104 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.192636 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.193534 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.193758 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.193843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.196446 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:58813: BlockingRecv error: recv got EOF from 127.4.58.67:58813 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:31:21 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:21 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:21.818944 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:21.813640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:42387)
0620 14:31:21.813910 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:21.813927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:21.814033 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:21.814310 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:21.814313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:21.814502 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:21.814648 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.814654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.815405 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.815408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.815974 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:21.815983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.816423 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.816426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:21.816509 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.817018 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:21.817038 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:21.818791 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
W20260620 14:31:21.819072 13505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:21.813728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:42387 (local address 127.4.58.126:44693)
0620 14:31:21.813969 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:21.813973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:21.813988 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:31:21.814126 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:21.814129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:21.814181 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:21.814268 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:21.814766 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.815300 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.816092 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:21.816323 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:21.816407 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:21.818943 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:42387: BlockingRecv error: recv got EOF from 127.4.58.65:42387 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:22.010928 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.005350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:46525)
0620 14:31:22.005624 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:22.005640 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:22.005760 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:22.006025 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:22.006027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:22.006261 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:22.006381 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.006390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.007143 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.007146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.007715 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.007723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.008196 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.008200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.008315 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.008885 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:22.008898 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:22.010799 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
W20260620 14:31:22.011058 13505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:22.005414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:46525 (local address 127.4.58.126:44693)
0620 14:31:22.005540 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:22.005544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:22.005721 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:31:22.005862 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:22.005864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:22.005913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:22.005986 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:22.006496 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.007035 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.007821 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.008061 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.008144 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.010942 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:46525: BlockingRecv error: recv got EOF from 127.4.58.66:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:22.202984 13507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.197359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:49891)
0620 14:31:22.197735 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:22.197752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:22.197874 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:22.198152 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:22.198155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:22.198359 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:22.198457 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.198463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.199285 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.199318 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.199890 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.199898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.200313 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.200316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.200381 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.200946 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:22.200959 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:22.202866 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":111,"threads_started":1}
W20260620 14:31:22.203133 13505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:22.197445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:49891 (local address 127.4.58.126:44693)
0620 14:31:22.197575 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:22.197579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:22.197832 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:31:22.197969 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:22.197973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:22.198033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:22.198117 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:22.198575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.199155 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.200009 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.200226 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.200302 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.203007 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:49891: BlockingRecv error: recv got EOF from 127.4.58.67:49891 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:22 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:22.825268 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:22.819780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:50153)
0620 14:31:22.820197 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:22.820213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:22.820329 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:22.820655 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:22.820658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:22.820901 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:22.821000 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.821005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.821709 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.821711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.822290 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:22.822297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.822732 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.822734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:22.822790 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.823251 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:22.823261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:22.825149 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":84,"threads_started":1}
W20260620 14:31:22.825415 13509 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:22.819857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:50153 (local address 127.4.58.126:44693)
0620 14:31:22.820198 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:22.820201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:22.820282 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:31:22.820420 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:22.820424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:22.820481 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:22.820618 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:22.821097 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.821627 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.822381 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:22.822646 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:22.822724 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:22.825294 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:50153: BlockingRecv error: recv got EOF from 127.4.58.65:50153 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":89,"threads_started":1}
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:23.016899 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.011707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:48595)
0620 14:31:23.011985 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.011998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.012114 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.012321 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.012323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.012495 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:23.012600 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.012606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.013358 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.013361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.013918 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.013926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.014383 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.014386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.014448 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.014916 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:23.014927 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:23.016723 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
W20260620 14:31:23.017000 13509 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:23.011804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:48595 (local address 127.4.58.126:44693)
0620 14:31:23.011950 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.011955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.012069 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:31:23.012182 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.012184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.012227 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.012289 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.012714 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.013267 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.014050 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.014279 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.014366 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.016902 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:48595: BlockingRecv error: recv got EOF from 127.4.58.66:48595 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:23.209326 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.203745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:51199)
0620 14:31:23.204047 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.204064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.204183 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.204438 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.204441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.204691 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:23.204820 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.204826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.205634 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.205638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.206189 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.206197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206646 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.206731 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.207260 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:23.207272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:23.209105 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
W20260620 14:31:23.209421 13509 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:23.203888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:51199 (local address 127.4.58.126:44693)
0620 14:31:23.204057 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.204062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.204137 (+ 75us) server_negotiation.cc:407] Connection header received
0620 14:31:23.204271 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.204274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.204325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.204402 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.204937 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.205526 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206294 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.206540 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.206628 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.209314 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:51199: BlockingRecv error: recv got EOF from 127.4.58.67:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:23 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:23.832031 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:23.826105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:55377)
0620 14:31:23.826426 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:23.826439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:23.826536 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:23.826838 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:23.826840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:23.827012 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:23.827128 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.827133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.827892 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.827895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.828491 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:23.828502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.829003 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.829006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:23.829089 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.829605 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:23.829617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:23.831854 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
W20260620 14:31:23.832126 13513 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:23.826220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55377 (local address 127.4.58.126:44693)
0620 14:31:23.826463 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:23.826468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:23.826486 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:31:23.826665 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:23.826668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:23.826725 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:23.826810 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:23.827228 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.827754 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.828613 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:23.828862 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:23.828951 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:23.831991 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:55377: BlockingRecv error: recv got EOF from 127.4.58.65:55377 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":56,"threads_started":1}
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:24.023524 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.017746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:36517)
0620 14:31:24.018018 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:24.018033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:24.018130 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:24.018359 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:24.018361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:24.018543 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:24.018638 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.018644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.019424 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.019428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.020013 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.020021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.020538 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.020542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.020613 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.021228 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:24.021240 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:24.023358 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
W20260620 14:31:24.023679 13513 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:24.017926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:36517 (local address 127.4.58.126:44693)
0620 14:31:24.018089 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.018092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.018109 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:31:24.018205 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.018207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.018250 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.018330 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.018760 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.019304 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.020144 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.020380 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.020472 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.023558 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:36517: BlockingRecv error: recv got EOF from 127.4.58.66:36517 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:24.215875 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.210251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:49669)
0620 14:31:24.210504 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:24.210517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:24.210621 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:24.210913 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:24.210916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:24.211142 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:24.211244 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.211251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.212045 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.212048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.212644 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.212653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.213172 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.213175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.213240 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.213787 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:24.213799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:24.215727 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
W20260620 14:31:24.215981 13513 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:24.210343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:49669 (local address 127.4.58.126:44693)
0620 14:31:24.210469 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.210473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.210583 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:31:24.210706 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.210708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.210758 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.210836 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.211386 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.211914 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.212786 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.213039 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.213127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.215843 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:49669: BlockingRecv error: recv got EOF from 127.4.58.67:49669 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:24 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:24.843091 13516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:24.832906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:50299)
0620 14:31:24.833166 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:24.833180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:24.833281 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:24.833615 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:24.833617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:24.833801 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:24.833902 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.833908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.834749 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.834752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.835336 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:24.835345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.835951 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.835954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:24.836029 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.840909 (+ 4880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:24.840925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:24.842956 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
W20260620 14:31:24.843261 13517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:24.833046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:50299 (local address 127.4.58.126:44693)
0620 14:31:24.833313 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:24.833318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:24.833330 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:31:24.833374 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:24.833376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:24.833418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:24.833555 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:24.834056 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.834620 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.835542 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:24.835804 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:24.835909 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:24.843121 (+ 7212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:50299: BlockingRecv error: recv got EOF from 127.4.58.65:50299 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":62,"threads_started":1}
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:25.029568 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.024285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:51453)
0620 14:31:25.024540 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.024613 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.024733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.024996 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.024998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.025178 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.025277 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.025283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.026015 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.026018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.026593 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.026601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.027068 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.027071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.027139 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.027672 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.027684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.029432 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
W20260620 14:31:25.029675 13517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:25.024364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:51453 (local address 127.4.58.126:44693)
0620 14:31:25.024538 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.024543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.024695 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:31:25.024844 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.024846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.024894 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.024967 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.025376 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.025909 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.026705 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.026922 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.027004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.029573 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:51453: BlockingRecv error: recv got EOF from 127.4.58.66:51453 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.67@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:25.222241 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.216869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.67:46157)
0620 14:31:25.217133 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.217146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.217263 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.217486 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.217488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.217674 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.217769 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.217774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.218519 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.218522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.219132 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.219141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.219645 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.219648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.219713 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.220185 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.220198 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.222111 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260620 14:31:25.222339 13517 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:25.216970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46157 (local address 127.4.58.126:44693)
0620 14:31:25.217099 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.217103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.217227 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:31:25.217346 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.217348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.217399 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.217461 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.217879 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.218416 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.219256 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.219476 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.219610 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.222241 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.67:46157: BlockingRecv error: recv got EOF from 127.4.58.67:46157 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:25 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.65@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:25.849985 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:25.844078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.65:60235)
0620 14:31:25.844343 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:25.844357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:25.844447 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:25.844787 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:25.844790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:25.844964 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:25.845067 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.845074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.845888 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.845891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.846455 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:25.846464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.847007 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.847010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:25.847081 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.847583 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:25.847596 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:25.849864 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
W20260620 14:31:25.850183 13521 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:25.844251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:60235 (local address 127.4.58.126:44693)
0620 14:31:25.844529 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:25.844533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:25.844547 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:31:25.844590 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:25.844592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:25.844633 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:25.844723 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:25.845213 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.845775 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.846582 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:25.846900 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:25.847002 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:25.850030 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.65:60235: BlockingRecv error: recv got EOF from 127.4.58.65:60235 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
Jun 20 14:31:26 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:31:26 dist-test-slave-gqzk krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.58.66@KRBTEST.COM for oryx/127.4.58.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:31:26.036082 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.030396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44693 (local address 127.4.58.66:34655)
0620 14:31:26.030722 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:26.030738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:26.030846 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:26.031142 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:26.031145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:26.031370 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:26.031494 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.031502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.032486 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.032489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.033118 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.033129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.033554 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.033557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.033622 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.034138 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:26.034150 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:26.035943 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.58.126:44693: Server oryx/127.4.58.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":142,"threads_started":1}
W20260620 14:31:26.036212 13521 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:26.030474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34655 (local address 127.4.58.126:44693)
0620 14:31:26.030723 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:26.030728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:26.030796 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:31:26.030966 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:26.030969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:26.031023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:26.031100 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:26.031638 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.032377 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.033217 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.033436 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.033520 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.036104 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.58.66:34655: BlockingRecv error: recv got EOF from 127.4.58.66:34655 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260620 14:31:26.123816 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13046
I20260620 14:31:26.128659 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13198
I20260620 14:31:26.133387 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13331
I20260620 14:31:26.138300 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 12976
2026-06-20T14:31:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17937 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:26Z Disabled control of system clock
I20260620 14:31:26.166755 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46211
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:46211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:31:26.279446 13526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:26.279750 13526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:26.279786 13526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:26.284046 13526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:26.284137 13526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:26.284158 13526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:26.284181 13526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:26.284207 13526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:26.289435 13526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:46211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:46211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:26.290594 13526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:26.292029 13526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:26.300686 13532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.300686 13531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.300784 13526 server_base.cc:1034] running on GCE node
W20260620 14:31:26.300714 13534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.301630 13526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:26.302836 13526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:26.304028 13526 hybrid_clock.cc:648] HybridClock initialized: now 1781965886304004 us; error 36 us; skew 500 ppm
I20260620 14:31:26.306126 13526 webserver.cc:460] Webserver started at http://127.4.58.126:34431/ using document root <none> and password file <none>
I20260620 14:31:26.306730 13526 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:26.306799 13526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:26.307031 13526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:26.309270 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "d40db4aa5c3148bc894e2d9967986df9"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "507b7eb9c66a656af76c1b95184fd78a"
server_key_iv: "ef1b0f462c6802aeba8dbff72b8c6da0"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.309852 13526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "d40db4aa5c3148bc894e2d9967986df9"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "507b7eb9c66a656af76c1b95184fd78a"
server_key_iv: "ef1b0f462c6802aeba8dbff72b8c6da0"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.313915 13526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:31:26.316408 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.317798 13526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:26.317919 13526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "d40db4aa5c3148bc894e2d9967986df9"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "507b7eb9c66a656af76c1b95184fd78a"
server_key_iv: "ef1b0f462c6802aeba8dbff72b8c6da0"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.318027 13526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:26.345942 13526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:26.346727 13526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:26.346925 13526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:26.353595 13526 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:46211
I20260620 14:31:26.353632 13592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:46211 every 8 connection(s)
I20260620 14:31:26.354535 13526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:26.357939 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:26.363058 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13526
I20260620 14:31:26.363180 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:31:26.363444 4329 external_mini_cluster.cc:1428] Setting key 7a515493ec404f40dd4631bf3265fda0
I20260620 14:31:26.364377 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9: Bootstrap starting.
I20260620 14:31:26.367298 13593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:26.368273 13593 log.cc:824] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:26.370515 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9: No bootstrap required, opened a new log
I20260620 14:31:26.374102 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.365072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.4.58.126:46211)
0620 14:31:26.365817 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:26.365838 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:26.365887 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:31:26.366778 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:26.366842 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:26.367382 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:26.367798 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:26.368280 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.369684 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.370648 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.371173 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.371300 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.372219 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:26.372224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:26.372238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:26.372648 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:26.372707 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:26.372729 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:26.372840 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:26.373002 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:31:26.373282 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":106,"threads_started":1}
I20260620 14:31:26.374920 13593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } }
I20260620 14:31:26.375223 13593 raft_consensus.cc:374] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:26.375294 13593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d40db4aa5c3148bc894e2d9967986df9, State: Initialized, Role: FOLLOWER
I20260620 14:31:26.375728 13593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } }
I20260620 14:31:26.375864 13593 raft_consensus.cc:388] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:26.375928 13593 raft_consensus.cc:482] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:26.376029 13593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:26.377144 13593 raft_consensus.cc:504] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } }
I20260620 14:31:26.377494 13593 leader_election.cc:302] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d40db4aa5c3148bc894e2d9967986df9; no voters:
I20260620 14:31:26.377825 13593 leader_election.cc:288] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:26.377938 13598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:26.378208 13598 raft_consensus.cc:686] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [term 1 LEADER]: Becoming Leader. State: Replica: d40db4aa5c3148bc894e2d9967986df9, State: Running, Role: LEADER
I20260620 14:31:26.378510 13598 consensus_queue.cc:237] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } }
I20260620 14:31:26.379035 13593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:26.379993 13600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d40db4aa5c3148bc894e2d9967986df9. Latest consensus state: current_term: 1 leader_uuid: "d40db4aa5c3148bc894e2d9967986df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } } }
I20260620 14:31:26.380119 13600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:26.380436 13607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:26.381093 13599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d40db4aa5c3148bc894e2d9967986df9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d40db4aa5c3148bc894e2d9967986df9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 46211 } } }
I20260620 14:31:26.381266 13599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:26.383746 13607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:26.389369 13607 catalog_manager.cc:1347] Generated new cluster ID: d75407282cbb4ff6b586603f736ea12c
I20260620 14:31:26.389433 13607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:26.414458 13607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:26.415424 13607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:26.423892 13607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d40db4aa5c3148bc894e2d9967986df9: Generated new TSK 0
I20260620 14:31:26.424625 13607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:26.434340 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--builtin_ntp_servers=127.4.58.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:31:26.546597 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:26.546916 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:26.546958 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:26.551149 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:26.551214 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:26.551270 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:26.556118 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:26.557313 13617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:26.558772 13617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:26.569146 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.569139 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.569144 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.569542 13617 server_base.cc:1034] running on GCE node
I20260620 14:31:26.570004 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:26.570747 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:26.571928 13617 hybrid_clock.cc:648] HybridClock initialized: now 1781965886571916 us; error 49 us; skew 500 ppm
I20260620 14:31:26.573946 13617 webserver.cc:460] Webserver started at http://127.4.58.65:43229/ using document root <none> and password file <none>
I20260620 14:31:26.574558 13617 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:26.574631 13617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:26.574828 13617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:26.577085 13617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "918ce31536224c7789ebc527125c404b"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "c5ce6df32dd68024e44770fb136d3a77"
server_key_iv: "82ad8f6220d0e23408818f0434889cf2"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.577737 13617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "918ce31536224c7789ebc527125c404b"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "c5ce6df32dd68024e44770fb136d3a77"
server_key_iv: "82ad8f6220d0e23408818f0434889cf2"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.582120 13617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 14:31:26.584939 13631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.586321 13617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:31:26.586453 13617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "918ce31536224c7789ebc527125c404b"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "c5ce6df32dd68024e44770fb136d3a77"
server_key_iv: "82ad8f6220d0e23408818f0434889cf2"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.586578 13617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:26.601447 13617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:26.602170 13617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:26.602347 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:26.602936 13617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:26.603890 13617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:26.603938 13617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.603994 13617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:26.604022 13617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.612569 13617 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:45651
I20260620 14:31:26.612597 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:45651 every 8 connection(s)
I20260620 14:31:26.613552 13617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:26.621076 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13617
I20260620 14:31:26.621196 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:26.621503 4329 external_mini_cluster.cc:1428] Setting key efe447d907fcaa0ece6d5ad13947105d
I20260620 14:31:26.623647 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--builtin_ntp_servers=127.4.58.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:31:26.623739 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.615598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:50241 (local address 127.4.58.126:46211)
0620 14:31:26.615727 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:26.615731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:26.616905 (+ 1174us) server_negotiation.cc:407] Connection header received
0620 14:31:26.618543 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:26.618546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:26.618597 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:26.618680 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:26.619578 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.620114 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.621575 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.621884 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.622001 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.623271 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:26.623274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:26.623276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:26.623287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:26.623309 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:26.623316 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:26.623368 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:26.623544 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:31:26.623593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:31:26.625073 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.615991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46211 (local address 127.4.58.65:50241)
0620 14:31:26.616698 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:26.616774 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:26.618254 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:26.618839 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:26.618846 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:26.618931 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:26.619436 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.619449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.620265 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.620268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.621372 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.621383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.622065 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.622071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.622274 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.622948 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:26.622950 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:26.623014 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:26.623104 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:26.623389 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:26.623398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:26.623405 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:26.623553 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:31:26.623898 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260620 14:31:26.626252 13745 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46211
I20260620 14:31:26.626475 13745 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:26.626991 13745 heartbeater.cc:511] Master 127.4.58.126:46211 requested a full tablet report, sending...
I20260620 14:31:26.628430 13557 ts_manager.cc:194] Registered new tserver with Master: 918ce31536224c7789ebc527125c404b (127.4.58.65:45651)
I20260620 14:31:26.629618 13557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.65:50241
W20260620 14:31:26.734975 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:26.735258 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:26.735311 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:26.739351 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:26.739419 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:26.739494 13748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:26.744419 13748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:26.745570 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:26.746973 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:26.756662 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.756676 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.756790 13748 server_base.cc:1034] running on GCE node
W20260620 14:31:26.756738 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.757445 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:26.758122 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:26.759292 13748 hybrid_clock.cc:648] HybridClock initialized: now 1781965886759269 us; error 36 us; skew 500 ppm
I20260620 14:31:26.761219 13748 webserver.cc:460] Webserver started at http://127.4.58.66:39655/ using document root <none> and password file <none>
I20260620 14:31:26.761787 13748 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:26.761855 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:26.762060 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:26.764124 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "0e96a4d29f594efa8fac597efcc803a8"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "25a795a4bf15992911bce856029670bb"
server_key_iv: "e597d80924d36b317116677507220fb8"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.764662 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "0e96a4d29f594efa8fac597efcc803a8"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "25a795a4bf15992911bce856029670bb"
server_key_iv: "e597d80924d36b317116677507220fb8"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.768641 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:31:26.771394 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.772825 13748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:26.772956 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0e96a4d29f594efa8fac597efcc803a8"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "25a795a4bf15992911bce856029670bb"
server_key_iv: "e597d80924d36b317116677507220fb8"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.773072 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:26.793324 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:26.794049 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:26.794246 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:26.794842 13748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:26.795804 13748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:26.795850 13748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.795904 13748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:26.795934 13748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.804481 13748 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:38675
I20260620 14:31:26.804503 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:38675 every 8 connection(s)
I20260620 14:31:26.805405 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:26.810420 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13748
I20260620 14:31:26.810515 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:26.810817 4329 external_mini_cluster.cc:1428] Setting key 0f8dbf8e953fb3033b96c27c28bc5a91
I20260620 14:31:26.812775 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--builtin_ntp_servers=127.4.58.84:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:31:26.814754 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.807208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:51295 (local address 127.4.58.126:46211)
0620 14:31:26.807331 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:26.807335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:26.808417 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 14:31:26.809841 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:26.809845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:26.809904 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:26.809976 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:26.810927 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.811540 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.812797 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.813057 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.813151 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.814242 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:26.814247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:26.814250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:26.814266 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:26.814289 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:26.814299 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:26.814366 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:26.814529 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:31:26.814597 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:31:26.815750 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.807604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46211 (local address 127.4.58.66:51295)
0620 14:31:26.808267 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:26.808308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:26.809620 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:26.810170 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:26.810177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:26.810264 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:26.810782 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.810796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.811669 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.811671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.812664 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.812673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.813202 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.813206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.813359 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.813953 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:26.813955 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:26.813998 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:26.814067 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:26.814376 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:26.814387 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:26.814393 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:31:26.814522 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:31:26.814828 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260620 14:31:26.816982 13876 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46211
I20260620 14:31:26.817265 13876 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:26.817926 13876 heartbeater.cc:511] Master 127.4.58.126:46211 requested a full tablet report, sending...
I20260620 14:31:26.819136 13557 ts_manager.cc:194] Registered new tserver with Master: 0e96a4d29f594efa8fac597efcc803a8 (127.4.58.66:38675)
I20260620 14:31:26.819759 13557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.66:51295
W20260620 14:31:26.923985 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:26.924230 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:26.924266 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:26.928364 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:26.928426 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:26.928483 13879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:26.933225 13879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:46211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:26.934329 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:26.935676 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:26.945220 13879 server_base.cc:1034] running on GCE node
W20260620 14:31:26.945173 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.945236 13885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:26.945175 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:26.945827 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:26.946450 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:26.947608 13879 hybrid_clock.cc:648] HybridClock initialized: now 1781965886947587 us; error 34 us; skew 500 ppm
I20260620 14:31:26.949550 13879 webserver.cc:460] Webserver started at http://127.4.58.67:37253/ using document root <none> and password file <none>
I20260620 14:31:26.950168 13879 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:26.950233 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:26.950426 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:26.952616 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "899c6562ee8f4d10a2d9949b98407bfd"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "e554da237b265345670d2d6fe4b28540"
server_key_iv: "3e538b318f8a28d6e2f0143797aa2bb1"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.953178 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "899c6562ee8f4d10a2d9949b98407bfd"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "e554da237b265345670d2d6fe4b28540"
server_key_iv: "3e538b318f8a28d6e2f0143797aa2bb1"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.956864 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:31:26.959420 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.960790 13879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:26.960912 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "899c6562ee8f4d10a2d9949b98407bfd"
format_stamp: "Formatted at 2026-06-20 14:31:26 on dist-test-slave-gqzk"
server_key: "e554da237b265345670d2d6fe4b28540"
server_key_iv: "3e538b318f8a28d6e2f0143797aa2bb1"
server_key_version: "encryptionkey@0"
I20260620 14:31:26.961015 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:26.971499 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:26.972203 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:26.972380 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:26.973003 13879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:26.973951 13879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:26.974000 13879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.974054 13879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:26.974084 13879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:26.982749 13879 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34623
I20260620 14:31:26.982810 14006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34623 every 8 connection(s)
I20260620 14:31:26.983685 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:26.988905 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 13879
I20260620 14:31:26.989040 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:26.989305 4329 external_mini_cluster.cc:1428] Setting key cf7ef009510c796f4d270745ce98af6a
I20260620 14:31:26.993021 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.985588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:37425 (local address 127.4.58.126:46211)
0620 14:31:26.985717 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:26.985721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:26.986818 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 14:31:26.988198 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:26.988202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:26.988258 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:26.988334 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:26.989375 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.989994 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.991206 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.991426 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.991510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.992538 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:26.992542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:26.992545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:26.992555 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:26.992582 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:26.992589 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:26.992645 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:26.992814 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:31:26.992876 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:31:26.993904 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:26.985983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:46211 (local address 127.4.58.67:37425)
0620 14:31:26.986667 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:26.986710 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:26.987977 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:26.988517 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:26.988524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:26.988637 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:26.989239 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.989254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.990117 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.990120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.991093 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:26.991102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:26.991562 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:26.991566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:26.991738 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:26.992291 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:26.992292 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:26.992333 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:26.992401 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:26.992657 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:26.992665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:26.992671 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:31:26.992815 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:31:26.993155 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260620 14:31:26.994964 14007 heartbeater.cc:347] Connected to a master server at 127.4.58.126:46211
I20260620 14:31:26.995211 14007 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:26.995702 14007 heartbeater.cc:511] Master 127.4.58.126:46211 requested a full tablet report, sending...
I20260620 14:31:26.996799 13557 ts_manager.cc:194] Registered new tserver with Master: 899c6562ee8f4d10a2d9949b98407bfd (127.4.58.67:34623)
I20260620 14:31:26.997371 13557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.67:37425
I20260620 14:31:27.003209 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:31:27.006320 13596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:27.005340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.4.58.126:46211)
0620 14:31:27.005476 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.005480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.005699 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:31:27.005832 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.005835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.005877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.005942 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.006210 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46516: BlockingRecv error: recv got EOF from 127.0.0.1:46516 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:31:27.006806 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13617
I20260620 14:31:27.011687 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13748
I20260620 14:31:27.016444 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13879
I20260620 14:31:27.021047 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 13526
2026-06-20T14:31:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (882 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:31:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:27Z Disabled control of system clock
I20260620 14:31:27.049572 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38499
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:38499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:31:27.161196 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.161494 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.161535 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.165755 14018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:27.165827 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.165841 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.165858 14018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:27.165871 14018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:27.170874 14018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:38499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:27.172034 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.173480 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.182464 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.182509 14018 server_base.cc:1034] running on GCE node
W20260620 14:31:27.182478 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.182464 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.183398 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.184583 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.185755 14018 hybrid_clock.cc:648] HybridClock initialized: now 1781965887185733 us; error 40 us; skew 500 ppm
I20260620 14:31:27.187709 14018 webserver.cc:460] Webserver started at http://127.4.58.126:46687/ using document root <none> and password file <none>
I20260620 14:31:27.188294 14018 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.188357 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.188553 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.190706 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "1d99f9024a464efea53e41f941a7f389"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "6e303f5a9340c5a82d70b414c75e377a"
server_key_iv: "363cf63630d0ed97295e77cbcc3fd05c"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.191278 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "1d99f9024a464efea53e41f941a7f389"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "6e303f5a9340c5a82d70b414c75e377a"
server_key_iv: "363cf63630d0ed97295e77cbcc3fd05c"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.195076 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:27.197558 14032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.198814 14018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:27.198948 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "1d99f9024a464efea53e41f941a7f389"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "6e303f5a9340c5a82d70b414c75e377a"
server_key_iv: "363cf63630d0ed97295e77cbcc3fd05c"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.199055 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.203675 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.204327 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.204484 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.211059 14018 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:38499
I20260620 14:31:27.211117 14084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:38499 every 8 connection(s)
I20260620 14:31:27.212028 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:27.215351 14085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:27.215386 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14018
I20260620 14:31:27.215485 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:31:27.215783 4329 external_mini_cluster.cc:1428] Setting key 441a1570b96aef82075a9e3eed741d50
I20260620 14:31:27.221966 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.217103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.4.58.126:38499)
0620 14:31:27.217814 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.217830 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.217869 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:31:27.218929 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.218955 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.219359 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.219721 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.220635 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.220639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.220653 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:27.220664 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.220724 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:27.220740 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.220905 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.220986 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.221235 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":107,"threads_started":1}
I20260620 14:31:27.223483 14085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389: Bootstrap starting.
I20260620 14:31:27.225977 14085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:27.226732 14085 log.cc:824] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:27.228816 14085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389: No bootstrap required, opened a new log
I20260620 14:31:27.232826 14085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } }
I20260620 14:31:27.233096 14085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:27.233141 14085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d99f9024a464efea53e41f941a7f389, State: Initialized, Role: FOLLOWER
I20260620 14:31:27.233533 14085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } }
I20260620 14:31:27.233659 14085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:27.233705 14085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:27.233777 14085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:27.234709 14085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } }
I20260620 14:31:27.235023 14085 leader_election.cc:302] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d99f9024a464efea53e41f941a7f389; no voters:
I20260620 14:31:27.235306 14085 leader_election.cc:288] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:27.235430 14090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:27.235646 14090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [term 1 LEADER]: Becoming Leader. State: Replica: 1d99f9024a464efea53e41f941a7f389, State: Running, Role: LEADER
I20260620 14:31:27.235958 14090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } }
I20260620 14:31:27.236510 14085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:27.237715 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d99f9024a464efea53e41f941a7f389. Latest consensus state: current_term: 1 leader_uuid: "1d99f9024a464efea53e41f941a7f389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } } }
I20260620 14:31:27.237847 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:27.237715 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d99f9024a464efea53e41f941a7f389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d99f9024a464efea53e41f941a7f389" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38499 } } }
I20260620 14:31:27.238145 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389 [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:27.238417 14099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:27.241217 14099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:27.247191 14099 catalog_manager.cc:1347] Generated new cluster ID: ef14cd8119814895a528e66e3d4f27ff
I20260620 14:31:27.247260 14099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:27.256553 14099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:27.256911 14099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:27.273011 14099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d99f9024a464efea53e41f941a7f389: Generated new TSK 0
I20260620 14:31:27.273970 14099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:31:27.288726 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--builtin_ntp_servers=127.4.58.84:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:31:27.401229 14109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.401541 14109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.401604 14109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.405856 14109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.405920 14109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.406010 14109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:27.410815 14109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:27.411949 14109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.413398 14109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:27.423316 14109 server_base.cc:1034] running on GCE node
W20260620 14:31:27.423273 14115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.423255 14114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.423269 14117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.423893 14109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.424518 14109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.425699 14109 hybrid_clock.cc:648] HybridClock initialized: now 1781965887425666 us; error 41 us; skew 500 ppm
I20260620 14:31:27.427769 14109 webserver.cc:460] Webserver started at http://127.4.58.65:39227/ using document root <none> and password file <none>
I20260620 14:31:27.428349 14109 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.428426 14109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.428630 14109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.430801 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "77439d407ea64f5b9dada994eafe01aa"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "880319a00a2cab88a69f949f1841dfc1"
server_key_iv: "5093fc85426ad82e40e0f738fde671e8"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.431365 14109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "77439d407ea64f5b9dada994eafe01aa"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "880319a00a2cab88a69f949f1841dfc1"
server_key_iv: "5093fc85426ad82e40e0f738fde671e8"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.435235 14109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:27.438006 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.439388 14109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:27.439548 14109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "77439d407ea64f5b9dada994eafe01aa"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "880319a00a2cab88a69f949f1841dfc1"
server_key_iv: "5093fc85426ad82e40e0f738fde671e8"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.439651 14109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.444223 14109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.444873 14109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.445030 14109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.445569 14109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:27.446493 14109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.446539 14109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.446588 14109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.446614 14109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.457147 14109 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:43811
I20260620 14:31:27.457180 14236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:43811 every 8 connection(s)
I20260620 14:31:27.458191 14109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:27.464260 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.460006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:57059 (local address 127.4.58.126:38499)
0620 14:31:27.460147 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.460151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.461277 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 14:31:27.462650 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.462653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.462691 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.462776 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.463895 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.463898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.463900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:27.463909 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.463925 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:27.463933 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.464027 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.464116 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.464153 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:31:27.465209 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.460419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38499 (local address 127.4.58.65:57059)
0620 14:31:27.461133 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.461174 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.462410 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.462973 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.462980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.463077 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.463658 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:27.463659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.463702 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:27.463775 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.464003 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.464008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.464012 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.464149 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.464349 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260620 14:31:27.465333 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14109
I20260620 14:31:27.465440 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:27.465752 4329 external_mini_cluster.cc:1428] Setting key a229338a200681a28cb5beb5326bf5eb
I20260620 14:31:27.466411 14237 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38499
I20260620 14:31:27.466620 14237 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.467026 14237 heartbeater.cc:511] Master 127.4.58.126:38499 requested a full tablet report, sending...
I20260620 14:31:27.467623 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--builtin_ntp_servers=127.4.58.84:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:31:27.468545 14049 ts_manager.cc:194] Registered new tserver with Master: 77439d407ea64f5b9dada994eafe01aa (127.4.58.65:43811)
W20260620 14:31:27.579784 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.580049 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.580086 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.584203 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.584265 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.584326 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:27.589226 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:27.590389 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.591811 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.601579 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.601579 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.601725 14240 server_base.cc:1034] running on GCE node
W20260620 14:31:27.601634 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.602237 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.602945 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.604156 14240 hybrid_clock.cc:648] HybridClock initialized: now 1781965887604119 us; error 49 us; skew 500 ppm
I20260620 14:31:27.606076 14240 webserver.cc:460] Webserver started at http://127.4.58.66:34257/ using document root <none> and password file <none>
I20260620 14:31:27.606648 14240 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.606714 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.606899 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.609061 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "75c06137dfef49d19060423048aa5f86"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "ac9e6e51d6f0fb73393fd10c8b67e981"
server_key_iv: "0dd8e3bed44a65d2c2dc563fb1527a1d"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.609704 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "75c06137dfef49d19060423048aa5f86"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "ac9e6e51d6f0fb73393fd10c8b67e981"
server_key_iv: "0dd8e3bed44a65d2c2dc563fb1527a1d"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.613584 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:31:27.616091 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.617416 14240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:31:27.617568 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "75c06137dfef49d19060423048aa5f86"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "ac9e6e51d6f0fb73393fd10c8b67e981"
server_key_iv: "0dd8e3bed44a65d2c2dc563fb1527a1d"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.617671 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.622179 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.622787 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.622939 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.623447 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:27.624326 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.624372 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.624423 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.624449 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.632510 14240 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46321
I20260620 14:31:27.632586 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46321 every 8 connection(s)
I20260620 14:31:27.633443 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:27.634013 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14240
I20260620 14:31:27.634172 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:27.634479 4329 external_mini_cluster.cc:1428] Setting key 86b4447bfcdad1591315fb26a14dc3ab
I20260620 14:31:27.637459 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--builtin_ntp_servers=127.4.58.84:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:31:27.640208 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.635904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:41987 (local address 127.4.58.126:38499)
0620 14:31:27.636030 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.636035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.636907 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:31:27.638399 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.638402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.638444 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.638526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.639754 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.639758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.639761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:27.639773 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.639793 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:27.639800 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.639879 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.640011 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.640055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:31:27.641144 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.635986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38499 (local address 127.4.58.66:41987)
0620 14:31:27.636767 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.636820 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.638202 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.638693 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.638701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.638798 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.639436 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:27.639438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.639481 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:27.639551 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.639872 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.639879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.639886 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.640051 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.640297 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260620 14:31:27.642336 14368 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38499
I20260620 14:31:27.642635 14368 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.643050 14368 heartbeater.cc:511] Master 127.4.58.126:38499 requested a full tablet report, sending...
I20260620 14:31:27.644140 14049 ts_manager.cc:194] Registered new tserver with Master: 75c06137dfef49d19060423048aa5f86 (127.4.58.66:46321)
W20260620 14:31:27.749754 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:27.750032 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:27.750074 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:27.754130 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:27.754189 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:27.754248 14371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:27.759034 14371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:38499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:27.760107 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:27.761515 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:27.771373 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:27.771389 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.771433 14371 server_base.cc:1034] running on GCE node
W20260620 14:31:27.771389 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:27.772064 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:27.772806 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:27.773977 14371 hybrid_clock.cc:648] HybridClock initialized: now 1781965887773933 us; error 58 us; skew 500 ppm
I20260620 14:31:27.775914 14371 webserver.cc:460] Webserver started at http://127.4.58.67:42879/ using document root <none> and password file <none>
I20260620 14:31:27.776494 14371 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:27.776556 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:27.776772 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:27.778864 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "331a5aafacc843bb986d0634f43e8f0f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "1de0db3bcac90ba3740021e6d8189419"
server_key_iv: "4b3d984ab80f7a8d8fe86a93949f6a5f"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.779481 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "331a5aafacc843bb986d0634f43e8f0f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "1de0db3bcac90ba3740021e6d8189419"
server_key_iv: "4b3d984ab80f7a8d8fe86a93949f6a5f"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.783250 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:31:27.785830 14385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.787077 14371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:27.787202 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "331a5aafacc843bb986d0634f43e8f0f"
format_stamp: "Formatted at 2026-06-20 14:31:27 on dist-test-slave-gqzk"
server_key: "1de0db3bcac90ba3740021e6d8189419"
server_key_iv: "4b3d984ab80f7a8d8fe86a93949f6a5f"
server_key_version: "encryptionkey@0"
I20260620 14:31:27.787307 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:27.791795 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:27.792418 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:27.792567 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:27.793176 14371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:27.794157 14371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:27.794201 14371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.794240 14371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:27.794268 14371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:27.802345 14371 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34975
I20260620 14:31:27.802376 14498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34975 every 8 connection(s)
I20260620 14:31:27.803308 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:27.809432 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.805177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:60515 (local address 127.4.58.126:38499)
0620 14:31:27.805325 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.805329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.806440 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 14:31:27.807785 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.807788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.807832 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.807903 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.808990 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:27.808993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:27.808996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:31:27.809009 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:27.809028 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:31:27.809036 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:27.809123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:27.809235 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:31:27.809277 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:31:27.810254 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:27.805575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:38499 (local address 127.4.58.67:60515)
0620 14:31:27.806258 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:27.806334 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:27.807522 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:27.808077 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:27.808084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:27.808175 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:27.808701 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:31:27.808702 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:27.808745 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:31:27.808838 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:27.809106 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:27.809114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:27.809120 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:31:27.809273 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:31:27.809516 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
I20260620 14:31:27.811297 14499 heartbeater.cc:347] Connected to a master server at 127.4.58.126:38499
I20260620 14:31:27.811511 14499 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:27.811929 14499 heartbeater.cc:511] Master 127.4.58.126:38499 requested a full tablet report, sending...
I20260620 14:31:27.812906 14049 ts_manager.cc:194] Registered new tserver with Master: 331a5aafacc843bb986d0634f43e8f0f (127.4.58.67:34975)
I20260620 14:31:27.813560 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14371
I20260620 14:31:27.813706 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:27.814020 4329 external_mini_cluster.cc:1428] Setting key 37caf111e0e321895e2a0bccf232be33
I20260620 14:31:27.816026 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:31:27.818769 14088 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:31:27.817851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.4.58.126:38499)
0620 14:31:27.817988 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:27.817992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:27.818142 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:31:27.818274 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:27.818277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:27.818311 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:27.818371 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:27.818664 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34318: BlockingRecv error: recv got EOF from 127.0.0.1:34318 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:31:27.819098 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14109
I20260620 14:31:27.824316 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14240
I20260620 14:31:27.829152 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14371
I20260620 14:31:27.833858 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14018
2026-06-20T14:31:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (812 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:27 dist-test-slave-gqzk krb5kdc[14509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:27 dist-test-slave-gqzk krb5kdc[14509](info): set up 2 sockets
Jun 20 14:31:27 dist-test-slave-gqzk krb5kdc[14509](info): commencing operation
krb5kdc: starting...
W20260620 14:31:28.801431 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:31:28 dist-test-slave-gqzk krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:31:28.920189 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34255
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:29.028429 14522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:29.028712 14522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:29.028789 14522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:29.032696 14522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:31:29.032773 14522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:29.032802 14522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:29.032825 14522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:31:29.032853 14522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:31:29.038060 14522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:29.039101 14522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:29.040097 14522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:29.048362 14527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:29.048422 14522 server_base.cc:1034] running on GCE node
W20260620 14:31:29.048355 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.048355 14528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:29.049273 14522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:29.050371 14522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:29.051538 14522 hybrid_clock.cc:648] HybridClock initialized: now 1781965889051520 us; error 29 us; skew 500 ppm
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:29.054504 14522 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:31:29.055445 14522 webserver.cc:460] Webserver started at http://127.4.58.126:33155/ using document root <none> and password file <none>
I20260620 14:31:29.056083 14522 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:29.056155 14522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:29.056370 14522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:29.058485 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "21da2ff1101c4c7e987ff503469bdd0e"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "e6c34d2f4dd5681665e33261bfed7622"
server_key_iv: "7730af570a4bd5f25611dea0169bba7e"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.059051 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "21da2ff1101c4c7e987ff503469bdd0e"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "e6c34d2f4dd5681665e33261bfed7622"
server_key_iv: "7730af570a4bd5f25611dea0169bba7e"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.062886 14522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 14:31:29.065300 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.066598 14522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:31:29.066720 14522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "21da2ff1101c4c7e987ff503469bdd0e"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "e6c34d2f4dd5681665e33261bfed7622"
server_key_iv: "7730af570a4bd5f25611dea0169bba7e"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.066818 14522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:29.083716 14522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:29.086721 14522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:29.086903 14522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:29.093798 14522 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34255
I20260620 14:31:29.094071 14589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34255 every 8 connection(s)
I20260620 14:31:29.094827 14522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:31:29.096551 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14522
I20260620 14:31:29.096671 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:31:29.097011 4329 external_mini_cluster.cc:1428] Setting key cce9670567ff423c4fc9184b95c75c08
I20260620 14:31:29.097872 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.104629 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: Bootstrap starting.
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:29.106997 14590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.107849 14590 log.cc:824] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:29.110306 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: No bootstrap required, opened a new log
I20260620 14:31:29.112911 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.098481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.4.58.126:34255)
0620 14:31:29.098922 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.098941 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.098984 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:31:29.099911 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.099936 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.100391 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.100797 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.101477 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.102910 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.103790 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.104484 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.104657 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.107334 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.107341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.107358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.107389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.109506 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.110005 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.110011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.110017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.110165 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.110517 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.110520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.110522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.111047 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.111240 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.111549 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.111863 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
I20260620 14:31:29.114706 14590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } }
I20260620 14:31:29.114934 14590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.115000 14590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21da2ff1101c4c7e987ff503469bdd0e, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.115453 14590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } }
I20260620 14:31:29.115584 14590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:31:29.115646 14590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:31:29.115741 14590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.116629 14590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } }
I20260620 14:31:29.116983 14590 leader_election.cc:302] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21da2ff1101c4c7e987ff503469bdd0e; no voters:
I20260620 14:31:29.117323 14590 leader_election.cc:288] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:31:29.117406 14595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:29.117569 14595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [term 1 LEADER]: Becoming Leader. State: Replica: 21da2ff1101c4c7e987ff503469bdd0e, State: Running, Role: LEADER
I20260620 14:31:29.117868 14595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } }
I20260620 14:31:29.118628 14590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:31:29.119520 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21da2ff1101c4c7e987ff503469bdd0e. Latest consensus state: current_term: 1 leader_uuid: "21da2ff1101c4c7e987ff503469bdd0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } } }
I20260620 14:31:29.119655 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:29.119948 14596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21da2ff1101c4c7e987ff503469bdd0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21da2ff1101c4c7e987ff503469bdd0e" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34255 } } }
I20260620 14:31:29.120042 14596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e [sys.catalog]: This master's current role is: LEADER
I20260620 14:31:29.120301 14604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:31:29.123566 14604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:31:29.129567 14604 catalog_manager.cc:1347] Generated new cluster ID: dc375043de8149728395cda7391f232e
I20260620 14:31:29.129649 14604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:31:29.156061 14604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:31:29.157089 14604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:31:29.162927 14604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: Generated new TSK 0
I20260620 14:31:29.163821 14604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:31:29.191821 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:29.301371 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:29.301669 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:29.301713 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:29.305850 14616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:29.305923 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:29.305986 14616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:31:29.311379 14616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:29.312495 14616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:29.313561 14616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:29.323800 14622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.323817 14624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.323856 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:29.323882 14616 server_base.cc:1034] running on GCE node
I20260620 14:31:29.324549 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:29.325232 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:29.326390 14616 hybrid_clock.cc:648] HybridClock initialized: now 1781965889326366 us; error 42 us; skew 500 ppm
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:29.329233 14616 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:31:29.330122 14616 webserver.cc:460] Webserver started at http://127.4.58.65:44939/ using document root <none> and password file <none>
I20260620 14:31:29.330685 14616 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:29.330750 14616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:29.330945 14616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:29.333066 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "bab6d4fd32d6dca3fa8bbf2aef30b526"
server_key_iv: "32041d676eb232e6ebd3566387fca734"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.333619 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "bab6d4fd32d6dca3fa8bbf2aef30b526"
server_key_iv: "32041d676eb232e6ebd3566387fca734"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.337415 14616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:31:29.339943 14631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.341377 14616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:29.341480 14616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "bab6d4fd32d6dca3fa8bbf2aef30b526"
server_key_iv: "32041d676eb232e6ebd3566387fca734"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.341593 14616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:29.353075 14616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:29.356010 14616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:29.356194 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:29.356798 14616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:29.357631 14616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:29.357678 14616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.357733 14616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:29.357757 14616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.366258 14616 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:35483
I20260620 14:31:29.366283 14744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:35483 every 8 connection(s)
I20260620 14:31:29.367173 14616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:31:29.368036 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14616
I20260620 14:31:29.368145 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:31:29.368403 4329 external_mini_cluster.cc:1428] Setting key 909cfed718fcf689d0a19500c51a9f0c
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:31:29.380927 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.368997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:36325 (local address 127.4.58.126:34255)
0620 14:31:29.369147 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.369150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.370139 (+ 989us) server_negotiation.cc:407] Connection header received
0620 14:31:29.371667 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.371669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.371716 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.371787 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.373208 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.373738 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.374869 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.375109 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.375198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.377601 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.377604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.377607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.377634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.378945 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.379394 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.379396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.379397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.379467 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.379814 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.379817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.379819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.380038 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.380132 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.380604 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.380735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:31:29.381965 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.369476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34255 (local address 127.4.58.65:36325)
0620 14:31:29.370004 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.370049 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.371409 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.371991 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.372001 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.372492 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:29.373064 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.373080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.373859 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.373863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.374754 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.374763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.375249 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.375253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.375425 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.375958 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:29.375974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:29.377439 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:29.379041 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.379046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.379058 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.379305 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.379549 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.379552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.379554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.379720 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.380135 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:29.380139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:29.380376 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.380541 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.380843 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260620 14:31:29.383302 14745 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34255
I20260620 14:31:29.383571 14745 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:29.384114 14745 heartbeater.cc:511] Master 127.4.58.126:34255 requested a full tablet report, sending...
I20260620 14:31:29.385679 14554 ts_manager.cc:194] Registered new tserver with Master: 99fe2fc6d57a4da990cfdfa0c9a1cd24 (127.4.58.65:35483)
I20260620 14:31:29.386849 14554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:36325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:31:29.396019 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:29.506227 14750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:29.506505 14750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:29.506549 14750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:29.510581 14750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:29.510653 14750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:29.510711 14750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:31:29.516085 14750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:29.517175 14750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:29.518165 14750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:31:29.527834 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.527823 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.527839 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:29.528034 14750 server_base.cc:1034] running on GCE node
I20260620 14:31:29.528564 14750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:29.529241 14750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:29.530380 14750 hybrid_clock.cc:648] HybridClock initialized: now 1781965889530351 us; error 48 us; skew 500 ppm
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:29.533034 14750 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:31:29.533875 14750 webserver.cc:460] Webserver started at http://127.4.58.66:43971/ using document root <none> and password file <none>
I20260620 14:31:29.534400 14750 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:29.534457 14750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:29.534650 14750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:29.536638 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "0f124a02a1434b57a3df5716391e2ea8"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "ee8915ddd0d66fb346f0aa760111a168"
server_key_iv: "49e5898c1358cf8e4af826dab95d484b"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.537212 14750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "0f124a02a1434b57a3df5716391e2ea8"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "ee8915ddd0d66fb346f0aa760111a168"
server_key_iv: "49e5898c1358cf8e4af826dab95d484b"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.540869 14750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:29.543376 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.544866 14750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:29.544970 14750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0f124a02a1434b57a3df5716391e2ea8"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "ee8915ddd0d66fb346f0aa760111a168"
server_key_iv: "49e5898c1358cf8e4af826dab95d484b"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.545077 14750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:29.562649 14750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:29.565874 14750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:29.566042 14750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:29.566632 14750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:29.567447 14750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:29.567492 14750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.567554 14750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:29.567584 14750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.576261 14750 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:38563
I20260620 14:31:29.576288 14878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:38563 every 8 connection(s)
I20260620 14:31:29.577276 14750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:31:29.582438 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14750
I20260620 14:31:29.582623 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:31:29.582890 4329 external_mini_cluster.cc:1428] Setting key c4a33ff7fafc45996cda805c2b3b8b42
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:29.591485 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.579162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34233 (local address 127.4.58.126:34255)
0620 14:31:29.579332 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.579336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.580172 (+ 836us) server_negotiation.cc:407] Connection header received
0620 14:31:29.581564 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.581567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.581624 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.581697 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.582902 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.583548 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.584793 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.585027 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.585117 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.587687 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.587690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.587692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.587719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.589235 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.589747 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.589750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.589751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.589833 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.590155 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.590158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.590160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.590349 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.590446 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.591248 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.591363 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:31:29.592911 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.579530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34255 (local address 127.4.58.66:34233)
0620 14:31:29.579997 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.580063 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.581343 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.581881 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.581889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.582253 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:29.582767 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.582782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.583665 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.583668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.584659 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.584668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.585164 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.585168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.585331 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.585923 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:29.585940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:29.587477 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:29.589348 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.589354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.589372 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.589649 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.589916 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.589919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.589921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.590068 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.590502 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:29.590510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:29.590934 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.591188 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.591601 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260620 14:31:29.594170 14879 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34255
I20260620 14:31:29.594514 14879 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:31:29.595156 14879 heartbeater.cc:511] Master 127.4.58.126:34255 requested a full tablet report, sending...
I20260620 14:31:29.596293 14554 ts_manager.cc:194] Registered new tserver with Master: 0f124a02a1434b57a3df5716391e2ea8 (127.4.58.66:38563)
I20260620 14:31:29.597009 14554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:34233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:31:29.609570 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:31:29.722645 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:31:29.722929 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:31:29.722975 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:31:29.727052 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:31:29.727114 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:31:29.727173 14884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:31:29.732518 14884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:31:29.733598 14884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:31:29.734689 14884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:31:29.744395 14884 server_base.cc:1034] running on GCE node
W20260620 14:31:29.744349 14892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.744349 14889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:31:29.744369 14890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:31:29.745390 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:31:29.746152 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:31:29.747326 14884 hybrid_clock.cc:648] HybridClock initialized: now 1781965889747303 us; error 34 us; skew 500 ppm
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:31:29.750321 14884 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:31:29.751281 14884 webserver.cc:460] Webserver started at http://127.4.58.67:45423/ using document root <none> and password file <none>
I20260620 14:31:29.751842 14884 fs_manager.cc:359] Metadata directory not provided
I20260620 14:31:29.751909 14884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:31:29.752102 14884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:31:29.754488 14884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "4de157ae219840e089b4a968325fb362"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "a06e15b9bb2444442662b5a394b39572"
server_key_iv: "e19eacbc9906abc3cd37c5de47569da5"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.755043 14884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "4de157ae219840e089b4a968325fb362"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "a06e15b9bb2444442662b5a394b39572"
server_key_iv: "e19eacbc9906abc3cd37c5de47569da5"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.759263 14884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 14:31:29.761907 14899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.763612 14884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:31:29.763754 14884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "4de157ae219840e089b4a968325fb362"
format_stamp: "Formatted at 2026-06-20 14:31:29 on dist-test-slave-gqzk"
server_key: "a06e15b9bb2444442662b5a394b39572"
server_key_iv: "e19eacbc9906abc3cd37c5de47569da5"
server_key_version: "encryptionkey@0"
I20260620 14:31:29.763887 14884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:31:29.780866 14884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:31:29.785123 14884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:31:29.785336 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:31:29.786046 14884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:31:29.787091 14884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:31:29.787146 14884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.787204 14884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:31:29.787230 14884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:31:29.798043 14884 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34913
I20260620 14:31:29.798082 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34913 every 8 connection(s)
I20260620 14:31:29.799039 14884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:31:29.806406 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 14884
I20260620 14:31:29.806519 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:31:29.806816 4329 external_mini_cluster.cc:1428] Setting key 8a443f93910e6e6e0c489f89be99bf58
Jun 20 14:31:29 dist-test-slave-gqzk krb5kdc[14509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965889, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:31:29.815567 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.801524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:44547 (local address 127.4.58.126:34255)
0620 14:31:29.801666 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.801670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.802780 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 14:31:29.804793 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.804798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.804856 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.804934 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.806754 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.807534 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.809195 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.809416 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.809498 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.812024 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.812027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.812029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.812055 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.813428 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.814040 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.814043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.814045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.814128 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.814447 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.814450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.814452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.814705 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.814795 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.815305 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.815411 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:31:29.816445 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.802025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34255 (local address 127.4.58.67:44547)
0620 14:31:29.802572 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.802656 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.804413 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.805162 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.805172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.805757 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:31:29.806512 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.806534 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.807695 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.807698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.809088 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.809101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.809595 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.809599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.809744 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.810297 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:31:29.810315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:31:29.811869 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:31:29.813631 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.813637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.813650 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.813929 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.814208 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:31:29.814210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:31:29.814212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:31:29.814346 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:31:29.814818 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:31:29.814823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:31:29.815071 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.815245 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.815513 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260620 14:31:29.817848 15013 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34255
I20260620 14:31:29.818102 15013 heartbeater.cc:464] Registering TS with master...
I20260620 14:31:29.818631 15013 heartbeater.cc:511] Master 127.4.58.126:34255 requested a full tablet report, sending...
I20260620 14:31:29.819731 14554 ts_manager.cc:194] Registered new tserver with Master: 4de157ae219840e089b4a968325fb362 (127.4.58.67:34913)
I20260620 14:31:29.820304 14554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:44547
I20260620 14:31:29.820858 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:31:29.830124 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.822984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.4.58.126:34255)
0620 14:31:29.823094 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.823098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.823297 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:31:29.823407 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.823410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.823454 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.823517 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:31:29.824179 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.824696 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.825536 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.825766 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.825850 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.827052 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:31:29.827055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:31:29.827058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:31:29.827085 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:31:29.828454 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.828928 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.828931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.828933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.829018 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:31:29.829267 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:31:29.829270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:31:29.829272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:31:29.829499 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:31:29.829616 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.829848 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.829958 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:31:29.833382 14553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:31:29.836303 14553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:31:29.850961 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.844988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38563 (local address 127.0.0.1:36834)
0620 14:31:29.845719 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.845772 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.845904 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.846902 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.846904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.846919 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.846984 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.846990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.848589 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.848593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.849610 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.849619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850425 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.850429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.850533 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.850568 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.850682 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.850808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":27,"spinlock_wait_cycles":8192,"thread_start_us":134,"threads_started":1}
I20260620 14:31:29.850967 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.845649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.4.58.66:38563)
0620 14:31:29.846452 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.846461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.846481 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:31:29.846542 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.846545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.846720 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.846913 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.847063 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.848330 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.849738 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850317 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.850413 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.850576 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.850711 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.850808 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":119,"threads_started":1}
I20260620 14:31:29.851333 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.844862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34913 (local address 127.0.0.1:46238)
0620 14:31:29.845720 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.845772 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.845904 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.846734 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.846740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.846761 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.846918 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.846926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.848389 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.848393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.849657 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.849667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850950 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.850953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.851059 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.851079 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.851146 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.851204 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":97,"threads_started":1}
I20260620 14:31:29.851752 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.845004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.4.58.67:34913)
0620 14:31:29.846182 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.846193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.846217 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:31:29.846293 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.846298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.846526 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.846682 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.847048 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.848252 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.849798 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850646 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.850751 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.850895 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.851509 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.851621 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":148,"threads_started":1}
I20260620 14:31:29.851773 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.845277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35483 (local address 127.0.0.1:49418)
0620 14:31:29.846114 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.846126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.846196 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.846803 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.846806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.846824 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.846944 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.846950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.848123 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.848127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.849657 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.849667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850777 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.850781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.850876 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.850898 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.850946 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.851016 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260620 14:31:29.851799 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.846021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.4.58.65:35483)
0620 14:31:29.846394 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.846407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.846426 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:31:29.846492 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.846497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.846666 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.847099 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.847224 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.847945 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.849807 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.850646 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.851364 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.851509 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.851557 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.851665 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":66,"threads_started":1}
I20260620 14:31:29.854339 14947 tablet_service.cc:1467] Processing CreateTablet for tablet 95e8c14617844373b75abae1aa869807 (DEFAULT_TABLE table=test-table [id=5bf66951caec49609360f07142b4a140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.854276 14813 tablet_service.cc:1467] Processing CreateTablet for tablet 95e8c14617844373b75abae1aa869807 (DEFAULT_TABLE table=test-table [id=5bf66951caec49609360f07142b4a140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.855266 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e8c14617844373b75abae1aa869807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.855268 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e8c14617844373b75abae1aa869807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.861248 15030 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Bootstrap starting.
I20260620 14:31:29.862092 15031 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Bootstrap starting.
I20260620 14:31:29.863644 14679 tablet_service.cc:1467] Processing CreateTablet for tablet 95e8c14617844373b75abae1aa869807 (DEFAULT_TABLE table=test-table [id=5bf66951caec49609360f07142b4a140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:31:29.864408 15031 tablet_bootstrap.cc:654] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.864717 14679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e8c14617844373b75abae1aa869807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:31:29.864897 15030 tablet_bootstrap.cc:654] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.865731 15031 log.cc:824] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:29.865731 15030 log.cc:824] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:29.867802 15031 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: No bootstrap required, opened a new log
I20260620 14:31:29.867827 15030 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: No bootstrap required, opened a new log
I20260620 14:31:29.868000 15031 ts_tablet_manager.cc:1397] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:31:29.868000 15030 ts_tablet_manager.cc:1397] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:31:29.871698 15034 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: Bootstrap starting.
I20260620 14:31:29.871912 15030 raft_consensus.cc:348] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.871912 15031 raft_consensus.cc:348] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.872195 15030 raft_consensus.cc:374] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.872195 15031 raft_consensus.cc:374] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.872251 15030 raft_consensus.cc:729] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f124a02a1434b57a3df5716391e2ea8, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.872251 15031 raft_consensus.cc:729] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de157ae219840e089b4a968325fb362, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.872721 15031 consensus_queue.cc:260] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.872714 15030 consensus_queue.cc:260] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.873651 14879 heartbeater.cc:503] Master 127.4.58.126:34255 was elected leader, sending a full tablet report...
I20260620 14:31:29.873651 15013 heartbeater.cc:503] Master 127.4.58.126:34255 was elected leader, sending a full tablet report...
I20260620 14:31:29.873770 15031 ts_tablet_manager.cc:1428] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:31:29.873770 15030 ts_tablet_manager.cc:1428] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:31:29.875053 15034 tablet_bootstrap.cc:654] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: Neither blocks nor log segments found. Creating new log.
I20260620 14:31:29.876369 15034 log.cc:824] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: Log is configured to *not* fsync() on all Append() calls
I20260620 14:31:29.878830 15034 tablet_bootstrap.cc:492] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: No bootstrap required, opened a new log
I20260620 14:31:29.879022 15034 ts_tablet_manager.cc:1397] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 14:31:29.882048 15034 raft_consensus.cc:348] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.882283 15034 raft_consensus.cc:374] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:31:29.882318 15034 raft_consensus.cc:729] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fe2fc6d57a4da990cfdfa0c9a1cd24, State: Initialized, Role: FOLLOWER
I20260620 14:31:29.882759 15034 consensus_queue.cc:260] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.883354 14745 heartbeater.cc:503] Master 127.4.58.126:34255 was elected leader, sending a full tablet report...
I20260620 14:31:29.883620 15034 ts_tablet_manager.cc:1428] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 14:31:29.908166 15038 raft_consensus.cc:482] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:31:29.908375 15038 raft_consensus.cc:504] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.909552 15038 leader_election.cc:288] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4de157ae219840e089b4a968325fb362 (127.4.58.67:34913), 0f124a02a1434b57a3df5716391e2ea8 (127.4.58.66:38563)
I20260620 14:31:29.913668 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.910009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:59609 (local address 127.4.58.66:38563)
0620 14:31:29.910206 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.910211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.910291 (+ 80us) server_negotiation.cc:407] Connection header received
0620 14:31:29.910505 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.910509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.910563 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.910610 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.911012 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.911549 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.912692 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.913127 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.913227 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.913286 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.913485 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.913556 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 14:31:29.913678 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.910290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:38997 (local address 127.4.58.67:34913)
0620 14:31:29.910404 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.910408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.910492 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:31:29.910602 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.910605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.910649 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.910720 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:31:29.911055 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.911543 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.912693 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.913116 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.913201 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.913264 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.913451 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.913557 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:31:29.913682 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.909853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34913 (local address 127.4.58.65:38997)
0620 14:31:29.910394 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.910408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.910508 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.910800 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.910803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.910822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.910942 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.910948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.911651 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.911655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.912566 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.912576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.913230 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.913234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.913335 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.913357 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.913416 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.913486 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":69,"thread_start_us":125,"threads_started":1}
I20260620 14:31:29.913690 15039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.909830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38563 (local address 127.4.58.65:59609)
0620 14:31:29.910205 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.910219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:31:29.910375 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:31:29.910751 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:31:29.910754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:31:29.910770 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:31:29.910886 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.910893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.911666 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.911670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.912560 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:31:29.912569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.913248 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.913252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:31:29.913359 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.913377 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:31:29.913459 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:31:29.913531 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260620 14:31:29.914499 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e8c14617844373b75abae1aa869807" candidate_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de157ae219840e089b4a968325fb362" is_pre_election: true
I20260620 14:31:29.914500 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e8c14617844373b75abae1aa869807" candidate_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f124a02a1434b57a3df5716391e2ea8" is_pre_election: true
I20260620 14:31:29.914789 14967 raft_consensus.cc:2454] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fe2fc6d57a4da990cfdfa0c9a1cd24 in term 0.
I20260620 14:31:29.914793 14833 raft_consensus.cc:2454] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fe2fc6d57a4da990cfdfa0c9a1cd24 in term 0.
I20260620 14:31:29.915230 14634 leader_election.cc:302] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4de157ae219840e089b4a968325fb362, 99fe2fc6d57a4da990cfdfa0c9a1cd24; no voters:
I20260620 14:31:29.915483 15038 raft_consensus.cc:2781] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:31:29.915567 15038 raft_consensus.cc:482] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:31:29.915614 15038 raft_consensus.cc:3037] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.916582 15038 raft_consensus.cc:504] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.916942 15038 leader_election.cc:288] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [CANDIDATE]: Term 1 election: Requested vote from peers 4de157ae219840e089b4a968325fb362 (127.4.58.67:34913), 0f124a02a1434b57a3df5716391e2ea8 (127.4.58.66:38563)
I20260620 14:31:29.917258 14967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e8c14617844373b75abae1aa869807" candidate_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4de157ae219840e089b4a968325fb362"
I20260620 14:31:29.917326 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e8c14617844373b75abae1aa869807" candidate_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f124a02a1434b57a3df5716391e2ea8"
I20260620 14:31:29.917403 14967 raft_consensus.cc:3037] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.917424 14833 raft_consensus.cc:3037] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:31:29.918454 14833 raft_consensus.cc:2454] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fe2fc6d57a4da990cfdfa0c9a1cd24 in term 1.
I20260620 14:31:29.918514 14967 raft_consensus.cc:2454] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fe2fc6d57a4da990cfdfa0c9a1cd24 in term 1.
I20260620 14:31:29.918718 14633 leader_election.cc:302] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f124a02a1434b57a3df5716391e2ea8, 99fe2fc6d57a4da990cfdfa0c9a1cd24; no voters:
I20260620 14:31:29.918875 15038 raft_consensus.cc:2781] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:31:29.919103 15038 raft_consensus.cc:686] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [term 1 LEADER]: Becoming Leader. State: Replica: 99fe2fc6d57a4da990cfdfa0c9a1cd24, State: Running, Role: LEADER
I20260620 14:31:29.919480 15038 consensus_queue.cc:237] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } }
I20260620 14:31:29.922900 14553 catalog_manager.cc:5495] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fe2fc6d57a4da990cfdfa0c9a1cd24 (127.4.58.65). New cstate: current_term: 1 leader_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99fe2fc6d57a4da990cfdfa0c9a1cd24" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35483 } health_report { overall_health: HEALTHY } } }
I20260620 14:31:29.935015 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.930904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.4.58.65:35483)
0620 14:31:29.931023 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:31:29.931027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:31:29.931042 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:31:29.931178 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:31:29.931181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:31:29.931239 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:31:29.931345 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:31:29.931603 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.932275 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.933491 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:31:29.933772 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:31:29.933859 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:31:29.934165 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:31:29.934640 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:31:29.934736 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:31:29.934841 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:31:29.934896 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:31:29.941233 14967 raft_consensus.cc:1261] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 1 FOLLOWER]: Refusing update from remote peer 99fe2fc6d57a4da990cfdfa0c9a1cd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:29.941233 14833 raft_consensus.cc:1261] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 1 FOLLOWER]: Refusing update from remote peer 99fe2fc6d57a4da990cfdfa0c9a1cd24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:31:29.942112 15041 consensus_queue.cc:1035] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4de157ae219840e089b4a968325fb362" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:29.942318 15038 consensus_queue.cc:1035] T 95e8c14617844373b75abae1aa869807 P 99fe2fc6d57a4da990cfdfa0c9a1cd24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f124a02a1434b57a3df5716391e2ea8" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:31:29.951051 15045 mvcc.cc:205] Tried to move back new op lower bound from 7298932285191090176 to 7298932285112913920. Current Snapshot: MvccSnapshot[applied={T|T < 7298932285191090176}]
I20260620 14:31:29.953751 15046 mvcc.cc:205] Tried to move back new op lower bound from 7298932285191090176 to 7298932285112913920. Current Snapshot: MvccSnapshot[applied={T|T < 7298932285191090176}]
I20260620 14:31:29.959260 14554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.959475 14554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:31:29.962224 14554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21da2ff1101c4c7e987ff503469bdd0e: Sending DeleteTablet for 3 replicas of tablet 95e8c14617844373b75abae1aa869807
I20260620 14:31:29.963186 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e8c14617844373b75abae1aa869807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:36834
I20260620 14:31:29.963224 14679 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e8c14617844373b75abae1aa869807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:49418
I20260620 14:31:29.963189 14947 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e8c14617844373b75abae1aa869807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:31:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:46238
I20260620 14:31:29.963861 15054 tablet_replica.cc:330] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: stopping tablet replica
I20260620 14:31:29.964239 15054 raft_consensus.cc:2229] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.964366 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14616
I20260620 14:31:29.964597 15054 raft_consensus.cc:2258] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.965034 15053 tablet_replica.cc:330] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: stopping tablet replica
I20260620 14:31:29.965355 15053 raft_consensus.cc:2229] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:31:29.965695 15053 raft_consensus.cc:2258] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:31:29.966338 15054 ts_tablet_manager.cc:1905] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:31:29.969172 15054 ts_tablet_manager.cc:1918] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:31:29.969264 15054 log.cc:1196] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/95e8c14617844373b75abae1aa869807
I20260620 14:31:29.969563 15054 ts_tablet_manager.cc:1939] T 95e8c14617844373b75abae1aa869807 P 4de157ae219840e089b4a968325fb362: Deleting consensus metadata
I20260620 14:31:29.972183 14540 catalog_manager.cc:4841] TS 4de157ae219840e089b4a968325fb362 (127.4.58.67:34913): tablet 95e8c14617844373b75abae1aa869807 (table test-table [id=5bf66951caec49609360f07142b4a140]) successfully deleted
I20260620 14:31:29.972506 15053 ts_tablet_manager.cc:1905] T 95e8c14617844373b75abae1aa869807 P 0f124a02a1434b57a3df5716391e2ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:31:29.972774 14541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:35483 (error 108)
I20260620 14:31:29.973408 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14750
I20260620 14:31:29.974079 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.973522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35483 (local address 127.0.0.1:49430)
0620 14:31:29.973918 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.973989 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
W20260620 14:31:29.974493 14541 catalog_manager.cc:4584] TS 99fe2fc6d57a4da990cfdfa0c9a1cd24 (127.4.58.65:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95e8c14617844373b75abae1aa869807: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35483: connect: Connection refused (error 111)
I20260620 14:31:29.978649 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14884
I20260620 14:31:29.980014 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:31:29.979707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38563 (local address 127.0.0.1:36840)
0620 14:31:29.979856 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:31:29.979919 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:38563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:31:29.983594 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 14522
2026-06-20T14:31:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2151 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:31:30 dist-test-slave-gqzk krb5kdc[15059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:31:30 dist-test-slave-gqzk krb5kdc[15059](info): set up 2 sockets
Jun 20 14:31:30 dist-test-slave-gqzk krb5kdc[15059](info): commencing operation
krb5kdc: starting...
W20260620 14:31:30.949536 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:31:31 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:31:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangeradmin_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangeradmin_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangerlookup_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangerlookup_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
I20260620 14:31:31.111186 4329 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCj3QH_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/postgres -l logfile start
2026-06-20 14:31:33.616 UTC [15089] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:31:33.616 UTC [15089] LOG: listening on IPv4 address "127.4.58.84", port 51211
2026-06-20 14:31:33.619 UTC [15089] LOG: listening on Unix socket "/tmp/.s.PGSQL.51211"
2026-06-20 14:31:33.647 UTC [15092] LOG: database system was shut down at 2026-06-20 14:31:32 UTC
2026-06-20 14:31:33.651 UTC [15089] LOG: database system is ready to accept connections
I20260620 14:31:34.529487 4329 mini_postgres.cc:96] Postgres bound to 51211
2026-06-20 14:31:34.535 UTC [15100] FATAL: database "slave" does not exist
127.4.58.84:51211 - accepting connections
I20260620 14:31:34.536198 4329 mini_ranger.cc:161] Starting Ranger...
I20260620 14:31:34.551045 4329 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:31:35.050268 4329 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:31:35.050346 4329 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin
2026-06-20 14:31:35,331 [I] DB FLAVOR :POSTGRES
2026-06-20 14:31:35,331 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:31:35,331 [I] Checking connection..
2026-06-20 14:31:35,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:31:35,586 [I] Checking connection passed.
2026-06-20 14:31:35,587 [I] --------- Verifying version history table ---------
2026-06-20 14:31:35,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:31:35,827 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:31:35,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:31:36,080 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:31:36,080 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:31:36,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:31:36.314 UTC [15164] WARNING: there is no transaction in progress
2026-06-20 14:31:36,320 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:31:36,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:31:36,564 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:31:36,565 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:31:36,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:31:36,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:31:37,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gqzk.c.gcp-upstream.internal','N') ;"
2026-06-20 14:31:37,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:31:37,550 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:31:37,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:31:37,791 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:31:37,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:31:38,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:31:38,269 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:31:38,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:31:39.246 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.262 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.280 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.296 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.312 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.329 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.385 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.393 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.408 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.418 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.429 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.443 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.453 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.460 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.468 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:39.475 UTC [15272] WARNING: there is no transaction in progress
2026-06-20 14:31:40,185 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:31:40,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:31:40,412 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:31:40,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:31:40,673 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:31:40,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:31:40,916 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:31:40,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:31:41,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:31:41,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gqzk.c.gcp-upstream.internal';"
2026-06-20 14:31:41,620 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:31:41,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:31:41,856 [I] DB_PATCHES have already been applied
I20260620 14:31:41.861840 4329 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCj3QH_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:31:41.861903 4329 mini_ranger.cc:193] Using host: 127.4.58.84
I20260620 14:31:41.865222 4329 mini_ranger.cc:227] Ranger admin URL: http://127.4.58.84:48867
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:31:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:31:42.850036 4329 mini_ranger.cc:160] Time spent starting Ranger: real 8.314s user 0.001s sys 0.009s
Jun 20, 2026 2:32:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:32:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:32:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:32:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:32:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:32:05.181016 4329 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangerkms_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/rangerkms_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.58.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.58.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:32:05 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:32:05.263360 4329 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:32:05.273379 4329 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:32:06.161621 4329 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:32:06.161726 4329 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms
I20260620 14:32:06.161832 4329 mini_ranger_kms.cc:227] Using postgres at 127.4.58.84:51211
2026-06-20 14:32:06,326 [I] DB FLAVOR :POSTGRES
2026-06-20 14:32:06,326 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:32:06,327 [I] Checking connection
2026-06-20 14:32:06,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:32:06,586 [I] connection success
2026-06-20 14:32:06,586 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:32:06,586 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:32:06,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:32:06,847 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:32:06,847 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:32:06,847 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:32:06,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:51211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:32:07,123 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:32:07.622956 4329 mini_ranger_kms.cc:310] Created kms service
I20260620 14:32:07.835258 4329 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:32:07.878635 4329 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:32:07 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965925, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:08.304174 4329 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:32:08.304356 4329 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:32:08.304388 4329 mini_ranger_kms.cc:240] Using host: 127.4.58.84
I20260620 14:32:08.307942 4329 mini_ranger_kms.cc:276] Ranger KMS PID: 15452
I20260620 14:32:08.308092 4329 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.58.84:47015
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:32:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:32:10.372946 4329 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.110s user 0.002s sys 0.147s
I20260620 14:32:10.373081 4329 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:32:10.373196 4329 mini_ranger_kms.cc:401] 127.4.58.84:47015/kms/v1/keys
Jun 20 14:32:10 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:32:10 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
Jun 20 14:32:11 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965930, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
Jun 20 14:32:13 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:32:13.555794 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34327
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34327
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:47015/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:13.667443 15503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:13.667749 15503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:13.667807 15503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:13.671974 15503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:13.672051 15503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:13.672077 15503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:13.672098 15503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:13.672117 15503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:13.677819 15503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.58.84:47015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34327
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:13.678957 15503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:13.679997 15503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:13.688967 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:13.689020 15503 server_base.cc:1034] running on GCE node
W20260620 14:32:13.689054 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:13.689054 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:13.689931 15503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:13.691136 15503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:13.692341 15503 hybrid_clock.cc:648] HybridClock initialized: now 1781965933692329 us; error 32 us; skew 500 ppm
Jun 20 14:32:13 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:13.699296 15503 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:32:13.700378 15503 webserver.cc:460] Webserver started at http://127.4.58.126:34271/ using document root <none> and password file <none>
I20260620 14:32:13.700981 15503 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:13.701042 15503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:13.701243 15503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:13 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:13.749267 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "8d49c8b928594c8e897726a05a625952"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-gqzk"
server_key: "e552f328d2a5765734e46ba004e926a8"
server_key_iv: "aabcd04b7ac41d0ca249fddf2474d922"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:13.750065 15503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "8d49c8b928594c8e897726a05a625952"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-gqzk"
server_key: "e552f328d2a5765734e46ba004e926a8"
server_key_iv: "aabcd04b7ac41d0ca249fddf2474d922"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:13.755451 15503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:32:13.773358 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:13.775264 15503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:32:13.775447 15503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "8d49c8b928594c8e897726a05a625952"
format_stamp: "Formatted at 2026-06-20 14:32:13 on dist-test-slave-gqzk"
server_key: "e552f328d2a5765734e46ba004e926a8"
server_key_iv: "aabcd04b7ac41d0ca249fddf2474d922"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:13.775609 15503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:13.803509 15503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:13.806677 15503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:13.806867 15503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:13.813862 15503 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34327
I20260620 14:32:13.813899 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34327 every 8 connection(s)
I20260620 14:32:13.814836 15503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:13.817948 15571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:13.823877 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 15503
I20260620 14:32:13.824014 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:32:13.824046 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: Bootstrap starting.
I20260620 14:32:13.824302 4329 external_mini_cluster.cc:1428] Setting key cf78d902f88f5c7d1ece418a2ec30c82
I20260620 14:32:13.826910 15571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:13.827833 15571 log.cc:824] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:13.830065 15571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: No bootstrap required, opened a new log
I20260620 14:32:13.834293 15571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } }
I20260620 14:32:13.834610 15571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:13.834815 15571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d49c8b928594c8e897726a05a625952, State: Initialized, Role: FOLLOWER
Jun 20 14:32:13 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:32:13.835399 15571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } }
I20260620 14:32:13.835568 15571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:13.835638 15571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:13.835750 15571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:13.836823 15571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } }
I20260620 14:32:13.837188 15571 leader_election.cc:302] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d49c8b928594c8e897726a05a625952; no voters:
I20260620 14:32:13.837587 15571 leader_election.cc:288] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:13.837682 15576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:13.837934 15576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [term 1 LEADER]: Becoming Leader. State: Replica: 8d49c8b928594c8e897726a05a625952, State: Running, Role: LEADER
I20260620 14:32:13.838733 15576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } }
I20260620 14:32:13.839087 15571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:13.841795 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:13.825795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.4.58.126:34327)
0620 14:32:13.826287 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:13.826306 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:13.826348 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:32:13.827195 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:13.827235 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:13.827687 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:13.828879 (+ 1192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:13.829351 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.830841 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.832116 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:13.832887 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:13.833057 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:13.836023 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:13.836029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:13.836048 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:13.836083 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:13.838163 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.838640 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.838646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.838653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.838796 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:13.839116 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:13.839119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:13.839120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:13.839638 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:13.839847 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:13.840633 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 14:32:13.840930 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260620 14:32:13.844014 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d49c8b928594c8e897726a05a625952. Latest consensus state: current_term: 1 leader_uuid: "8d49c8b928594c8e897726a05a625952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } } }
I20260620 14:32:13.844156 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:13.844975 15590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:13.845021 15571 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:32:13.846102 15577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d49c8b928594c8e897726a05a625952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d49c8b928594c8e897726a05a625952" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34327 } } }
I20260620 14:32:13.846271 15577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:13.851708 15590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:32:13.859433 15590 catalog_manager.cc:1347] Generated new cluster ID: dbf19bd4ddca4f2fa0f41e67281ae20e
I20260620 14:32:13.859512 15590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:13.896632 15590 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:13.898486 15590 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:13.903903 15590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: Generated new TSK 0
I20260620 14:32:13.904851 15590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:32:14.572649 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:14.712370 15616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:14.712915 15616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:14.713035 15616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:14.718240 15616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:14.718312 15616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:14.718394 15616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:32:14.725004 15616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.58.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:14.726722 15616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:14.728066 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:14.739082 15624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:14.739043 15623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:14.739382 15626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:14.739574 15616 server_base.cc:1034] running on GCE node
I20260620 14:32:14.740361 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:14.741194 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:14.742980 15616 hybrid_clock.cc:648] HybridClock initialized: now 1781965934742907 us; error 93 us; skew 500 ppm
Jun 20 14:32:14 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:14.746637 15616 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:32:14.747799 15616 webserver.cc:460] Webserver started at http://127.4.58.65:33945/ using document root <none> and password file <none>
I20260620 14:32:14.748531 15616 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:14.748625 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:14.748955 15616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:14 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:14.767560 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "b2afa944a2a94d559ff1164bea12e594"
format_stamp: "Formatted at 2026-06-20 14:32:14 on dist-test-slave-gqzk"
server_key: "57ddafdd12f10f983a404136949d2d0c"
server_key_iv: "ec37ebf8202ba658bb2189931ba26299"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:14.768327 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "b2afa944a2a94d559ff1164bea12e594"
format_stamp: "Formatted at 2026-06-20 14:32:14 on dist-test-slave-gqzk"
server_key: "57ddafdd12f10f983a404136949d2d0c"
server_key_iv: "ec37ebf8202ba658bb2189931ba26299"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:14.778489 15616 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260620 14:32:14.807529 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:14.809039 15616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:32:14.809175 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "b2afa944a2a94d559ff1164bea12e594"
format_stamp: "Formatted at 2026-06-20 14:32:14 on dist-test-slave-gqzk"
server_key: "57ddafdd12f10f983a404136949d2d0c"
server_key_iv: "ec37ebf8202ba658bb2189931ba26299"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:14.809306 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:14.825090 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:14.828562 15616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:14.828825 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:14.829546 15616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:14.830665 15616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:14.830787 15616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:14.830925 15616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:14.831009 15616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:14.841655 15616 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:43105
I20260620 14:32:14.842993 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:14.850292 15748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:43105 every 8 connection(s)
I20260620 14:32:14.851943 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 15616
I20260620 14:32:14.852077 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:14.852391 4329 external_mini_cluster.cc:1428] Setting key 7df785f738db25b2106a6b1cbeb70726
Jun 20 14:32:14 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:32:14.869638 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:14.853054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:60543 (local address 127.4.58.126:34327)
0620 14:32:14.854616 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:14.854626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:14.856256 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 14:32:14.856305 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:14.856308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:14.856365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:14.856445 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:14.857998 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:14.858779 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:14.860956 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:14.861313 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:14.861395 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:14.864345 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:14.864350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:14.864353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:14.864382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:14.865967 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:14.866505 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:14.866508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:14.866510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:14.866589 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:14.867039 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:14.867042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:14.867043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:14.868458 (+ 1415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:14.868553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:14.869324 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 14:32:14.869457 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":117,"threads_started":1}
I20260620 14:32:14.871683 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:14.853502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34327 (local address 127.4.58.65:60543)
0620 14:32:14.854900 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:14.854961 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:14.855958 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:14.856783 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:14.856793 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:14.857146 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:14.857848 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:14.857864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:14.858920 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:14.858923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:14.860807 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:14.860820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:14.861614 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:14.861617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:14.861822 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:14.862526 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:14.862548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:14.864120 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:14.866135 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:14.866140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:14.866153 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:14.866398 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:14.866714 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:14.866717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:14.866719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:14.866932 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:14.868608 (+ 1676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:14.868615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:14.869031 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 14:32:14.870181 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0620 14:32:14.870558 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":156,"threads_started":1}
I20260620 14:32:14.873150 15749 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34327
I20260620 14:32:14.873404 15749 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:14.873874 15749 heartbeater.cc:511] Master 127.4.58.126:34327 requested a full tablet report, sending...
I20260620 14:32:14.875569 15534 ts_manager.cc:194] Registered new tserver with Master: b2afa944a2a94d559ff1164bea12e594 (127.4.58.65:43105)
I20260620 14:32:14.877130 15534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:60543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:32:14.886193 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:32:14 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:32:15.013319 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:15.013785 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:15.013912 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:15.019035 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:15.019111 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:15.019193 15755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:32:15.025192 15755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.58.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:15.026300 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:15.027328 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:15.036643 15762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:15.037384 15755 server_base.cc:1034] running on GCE node
W20260620 14:32:15.037536 15761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:15.037679 15764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:15.038251 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:15.039091 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:15.040316 15755 hybrid_clock.cc:648] HybridClock initialized: now 1781965935040285 us; error 46 us; skew 500 ppm
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:15.044112 15755 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:32:15.045428 15755 webserver.cc:460] Webserver started at http://127.4.58.66:34877/ using document root <none> and password file <none>
I20260620 14:32:15.046214 15755 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:15.046334 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:15.046645 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:15.064350 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "1b22bb5f639f43c2819ba25eadbc50c0"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "8f8c3ab11deb4e4e6b577e58c4d134f8"
server_key_iv: "fb925ae5bed9704cae944438a08fd434"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.065287 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "1b22bb5f639f43c2819ba25eadbc50c0"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "8f8c3ab11deb4e4e6b577e58c4d134f8"
server_key_iv: "fb925ae5bed9704cae944438a08fd434"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.070489 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 14:32:15.092049 15771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.094868 15755 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260620 14:32:15.095089 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "1b22bb5f639f43c2819ba25eadbc50c0"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "8f8c3ab11deb4e4e6b577e58c4d134f8"
server_key_iv: "fb925ae5bed9704cae944438a08fd434"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.095296 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:15.127676 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:15.131554 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:15.131806 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:15.132580 15755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:15.133759 15755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:15.133879 15755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.133985 15755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:15.134044 15755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.144944 15755 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:38111
I20260620 14:32:15.146199 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:32:15.152138 15884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:38111 every 8 connection(s)
I20260620 14:32:15.155290 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 15755
I20260620 14:32:15.155449 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:32:15.155769 4329 external_mini_cluster.cc:1428] Setting key a5a6109b37c16464417d5472eefb1ed2
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:32:15.171591 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.156330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:34403 (local address 127.4.58.126:34327)
0620 14:32:15.156447 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.156452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.156885 (+ 433us) server_negotiation.cc:407] Connection header received
0620 14:32:15.159623 (+ 2738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.159627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.159690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.159764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:15.161561 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.162435 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.164220 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.164575 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.164662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.167310 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:15.167313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:15.167315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:15.167343 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:15.168957 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.169682 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.169685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.169687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.169785 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.170227 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.170231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.170233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.170458 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:15.170549 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.171238 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.171400 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:32:15.172626 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.154875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34327 (local address 127.4.58.66:34403)
0620 14:32:15.156679 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.156781 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.159350 (+ 2569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.160220 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.160230 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.160611 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:15.161380 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.161403 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.162611 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.162615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.164064 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.164077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.164926 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.164930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.165151 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.165757 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:15.165775 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:15.167121 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:15.169134 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:15.169141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:15.169162 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:15.169552 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:15.169900 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:15.169903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:15.169905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:15.170113 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:15.170559 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:15.170567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:15.170925 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.171184 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.171638 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":170,"threads_started":1}
I20260620 14:32:15.174054 15885 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34327
I20260620 14:32:15.174340 15885 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:15.174916 15885 heartbeater.cc:511] Master 127.4.58.126:34327 requested a full tablet report, sending...
I20260620 14:32:15.176158 15534 ts_manager.cc:194] Registered new tserver with Master: 1b22bb5f639f43c2819ba25eadbc50c0 (127.4.58.66:38111)
I20260620 14:32:15.176971 15534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:34403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:32:15.188360 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:47015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
W20260620 14:32:15.350556 15890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:15.350973 15890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:15.351074 15890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:15.357345 15890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:15.357498 15890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:15.357594 15890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:32:15.365558 15890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.58.84:47015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:15.367286 15890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:15.368676 15890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:15.381188 15896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:15.381304 15898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:15.381680 15895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:15.382781 15890 server_base.cc:1034] running on GCE node
I20260620 14:32:15.383401 15890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:15.384219 15890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:15.385553 15890 hybrid_clock.cc:648] HybridClock initialized: now 1781965935385456 us; error 108 us; skew 500 ppm
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:15.389362 15890 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:32:15.390625 15890 webserver.cc:460] Webserver started at http://127.4.58.67:38697/ using document root <none> and password file <none>
I20260620 14:32:15.391208 15890 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:15.391276 15890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:15.391512 15890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:15.415376 15890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "91842a65d5344ed0b0b9ad2dc111f7a3"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "387f52dc33d4d82411988acadbf3d10d"
server_key_iv: "bea3f9c0aa82745e9ceaa95a17530a3b"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.419819 15890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "91842a65d5344ed0b0b9ad2dc111f7a3"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "387f52dc33d4d82411988acadbf3d10d"
server_key_iv: "bea3f9c0aa82745e9ceaa95a17530a3b"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.426067 15890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 14:32:15.465205 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.467046 15890 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 14:32:15.467190 15890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "91842a65d5344ed0b0b9ad2dc111f7a3"
format_stamp: "Formatted at 2026-06-20 14:32:15 on dist-test-slave-gqzk"
server_key: "387f52dc33d4d82411988acadbf3d10d"
server_key_iv: "bea3f9c0aa82745e9ceaa95a17530a3b"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:15.467324 15890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:15.498905 15890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:15.502926 15890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:15.503187 15890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:15.503959 15890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:15.505141 15890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:15.505304 15890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.505411 15890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:15.505481 15890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:15.516233 15890 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:41987
I20260620 14:32:15.520649 15890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:32:15.521852 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 15890
I20260620 14:32:15.521989 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:32:15.522305 4329 external_mini_cluster.cc:1428] Setting key 125578f619fef20e3bb2a0e0f1d9fb27
I20260620 14:32:15.522706 16018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:41987 every 8 connection(s)
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965935, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:32:15.548923 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.526572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:44953 (local address 127.4.58.126:34327)
0620 14:32:15.526696 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.526700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.533463 (+ 6763us) server_negotiation.cc:407] Connection header received
0620 14:32:15.534812 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.534816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.534877 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.534942 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:15.536736 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.537632 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.539460 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.539812 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.539898 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.544356 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:15.544361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:15.544364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:15.544408 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:15.546250 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.547008 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.547012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.547014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.547114 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.547544 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.547547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.547549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.547786 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:15.547856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.548603 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.548765 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:32:15.550812 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.525439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34327 (local address 127.4.58.67:44953)
0620 14:32:15.533194 (+ 7755us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.533288 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.534497 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.535431 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.535443 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.535820 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:15.536559 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.536580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.537790 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.537794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.539323 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.539334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.540113 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.540161 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.540380 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.542283 (+ 1903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:15.542307 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:15.544108 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:15.546438 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:15.546445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:15.546467 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:15.546872 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:15.547221 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:15.547224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:15.547226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:15.547435 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:15.547938 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:15.547946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:15.548315 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.549370 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.549755 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7437,"thread_start_us":144,"threads_started":1}
I20260620 14:32:15.552258 16019 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34327
I20260620 14:32:15.552507 16019 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:15.553081 16019 heartbeater.cc:511] Master 127.4.58.126:34327 requested a full tablet report, sending...
I20260620 14:32:15.554241 15534 ts_manager.cc:194] Registered new tserver with Master: 91842a65d5344ed0b0b9ad2dc111f7a3 (127.4.58.67:41987)
I20260620 14:32:15.554968 15534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:44953
I20260620 14:32:15.560663 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:32:15.573351 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.563002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.4.58.126:34327)
0620 14:32:15.563112 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.563116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.563734 (+ 618us) server_negotiation.cc:407] Connection header received
0620 14:32:15.563915 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.563918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.563967 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.564014 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:15.565274 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.566083 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.567276 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.567632 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.567708 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.569568 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:15.569572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:15.569575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:15.569602 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:15.571333 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.571845 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.571849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.571851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.571937 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:15.572222 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:15.572225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:15.572227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:15.572464 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:15.572571 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.573055 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.573181 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:32:15.577280 15534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:32:15.581744 15534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:32:15.607092 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.596964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:41987 (local address 127.0.0.1:57894)
0620 14:32:15.599629 (+ 2665us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.599639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.599697 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.601315 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.601322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.601346 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:15.601493 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.601499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.602916 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.602920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.605176 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.605187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.606499 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.606504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.606631 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.606679 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.606797 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.606932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2621,"thread_start_us":86,"threads_started":1}
I20260620 14:32:15.608562 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.597322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.4.58.67:41987)
0620 14:32:15.599255 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.599267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.599776 (+ 509us) server_negotiation.cc:407] Connection header received
0620 14:32:15.599861 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.599866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.600106 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.600251 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:15.601618 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.602786 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.605322 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.606150 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.606251 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.606432 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.608269 (+ 1837us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.608407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":123,"threads_started":1}
I20260620 14:32:15.612443 15953 tablet_service.cc:1467] Processing CreateTablet for tablet f5c885267c854922bcd68e47c9fb75e2 (DEFAULT_TABLE table=test-table [id=e2d0b45a817b4b30b8ef67d0539288bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:15.614642 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c885267c854922bcd68e47c9fb75e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:15.624002 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.596345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38111 (local address 127.0.0.1:51524)
0620 14:32:15.599401 (+ 3056us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.599414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.599490 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.614266 (+ 14776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.614270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.614289 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:15.614419 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.614425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.618990 (+ 4565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.618994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.620143 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.620154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.623588 (+ 3434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.623592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.623685 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.623716 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.623763 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.623841 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2988,"thread_start_us":94,"threads_started":1}
I20260620 14:32:15.624923 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.596522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.4.58.66:38111)
0620 14:32:15.613033 (+ 16511us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.613050 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.613077 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:32:15.613161 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.613167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.613396 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.614578 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:15.614721 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.618882 (+ 4161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.620316 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.621178 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.624439 (+ 3261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.624610 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.624662 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.624782 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16381,"thread_start_us":107,"threads_started":1}
I20260620 14:32:15.625576 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.594211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43105 (local address 127.0.0.1:43042)
0620 14:32:15.598776 (+ 4565us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.598837 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.598962 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.617209 (+ 18247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.617213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.617234 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:15.617328 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.617334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.621313 (+ 3979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.621316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.622414 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.622425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.625209 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.625213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.625300 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.625324 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.625381 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.625451 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4367,"thread_start_us":81,"threads_started":1}
I20260620 14:32:15.626504 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.594374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.4.58.65:43105)
0620 14:32:15.613713 (+ 19339us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.613720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.613742 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:32:15.613814 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.613819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.614103 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.617517 (+ 3414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:15.617654 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.618679 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.622573 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.623434 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.626029 (+ 2595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.626208 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.626261 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.626368 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19225,"thread_start_us":124,"threads_started":1}
I20260620 14:32:15.625612 16038 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Bootstrap starting.
I20260620 14:32:15.629691 16038 tablet_bootstrap.cc:654] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:15.629762 15808 tablet_service.cc:1467] Processing CreateTablet for tablet f5c885267c854922bcd68e47c9fb75e2 (DEFAULT_TABLE table=test-table [id=e2d0b45a817b4b30b8ef67d0539288bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:15.630636 16038 log.cc:824] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:15.630892 15808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c885267c854922bcd68e47c9fb75e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:15.632786 16038 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: No bootstrap required, opened a new log
I20260620 14:32:15.632980 16038 ts_tablet_manager.cc:1397] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 14:32:15.638186 16038 raft_consensus.cc:348] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.638629 16038 raft_consensus.cc:374] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:15.638742 16038 raft_consensus.cc:729] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91842a65d5344ed0b0b9ad2dc111f7a3, State: Initialized, Role: FOLLOWER
I20260620 14:32:15.639442 16038 consensus_queue.cc:260] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.640702 16038 ts_tablet_manager.cc:1428] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 14:32:15.641117 16019 heartbeater.cc:503] Master 127.4.58.126:34327 was elected leader, sending a full tablet report...
I20260620 14:32:15.642069 15683 tablet_service.cc:1467] Processing CreateTablet for tablet f5c885267c854922bcd68e47c9fb75e2 (DEFAULT_TABLE table=test-table [id=e2d0b45a817b4b30b8ef67d0539288bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:32:15.643215 15683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c885267c854922bcd68e47c9fb75e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:32:15.653163 16042 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Bootstrap starting.
I20260620 14:32:15.653884 16040 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Bootstrap starting.
I20260620 14:32:15.655804 16042 tablet_bootstrap.cc:654] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:15.656558 16040 tablet_bootstrap.cc:654] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:15.656833 16042 log.cc:824] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:15.657567 16040 log.cc:824] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:15.659381 16042 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: No bootstrap required, opened a new log
I20260620 14:32:15.659584 16042 ts_tablet_manager.cc:1397] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:32:15.663938 16042 raft_consensus.cc:348] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.664285 16042 raft_consensus.cc:374] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:15.664336 16042 raft_consensus.cc:729] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2afa944a2a94d559ff1164bea12e594, State: Initialized, Role: FOLLOWER
I20260620 14:32:15.665007 16042 consensus_queue.cc:260] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.666209 16042 ts_tablet_manager.cc:1428] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:32:15.666630 15749 heartbeater.cc:503] Master 127.4.58.126:34327 was elected leader, sending a full tablet report...
I20260620 14:32:15.673738 16040 tablet_bootstrap.cc:492] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: No bootstrap required, opened a new log
I20260620 14:32:15.673972 16040 ts_tablet_manager.cc:1397] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20260620 14:32:15.679922 16040 raft_consensus.cc:348] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.680274 16040 raft_consensus.cc:374] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:15.680337 16040 raft_consensus.cc:729] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b22bb5f639f43c2819ba25eadbc50c0, State: Initialized, Role: FOLLOWER
I20260620 14:32:15.680905 16040 consensus_queue.cc:260] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.682018 16040 ts_tablet_manager.cc:1428] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 14:32:15.682494 15885 heartbeater.cc:503] Master 127.4.58.126:34327 was elected leader, sending a full tablet report...
Jun 20 14:32:15 dist-test-slave-gqzk krb5kdc[15059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965934, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
W20260620 14:32:15.773418 16020 tablet.cc:2369] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:32:15.854511 15750 tablet.cc:2369] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:32:15.859581 16041 raft_consensus.cc:482] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:32:15.859813 16041 raft_consensus.cc:504] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.861313 16041 leader_election.cc:288] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2afa944a2a94d559ff1164bea12e594 (127.4.58.65:43105), 1b22bb5f639f43c2819ba25eadbc50c0 (127.4.58.66:38111)
I20260620 14:32:15.866755 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.861515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43105 (local address 127.4.58.67:38341)
0620 14:32:15.862038 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.862054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.862148 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.862796 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.862799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.862822 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:15.862953 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.862960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.864040 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.864044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.865355 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.865367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.866336 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.866340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.866445 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.866468 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.866537 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.866613 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446}
I20260620 14:32:15.867242 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.862413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:38341 (local address 127.4.58.65:43105)
0620 14:32:15.862557 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.862561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.862577 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:32:15.862629 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.862632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.862676 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.862757 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:15.863097 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.863905 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.865522 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.866100 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.866206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.866276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.867045 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.867125 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:32:15.867846 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c885267c854922bcd68e47c9fb75e2" candidate_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afa944a2a94d559ff1164bea12e594" is_pre_election: true
I20260620 14:32:15.868203 15700 raft_consensus.cc:2454] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91842a65d5344ed0b0b9ad2dc111f7a3 in term 0.
I20260620 14:32:15.868805 15908 leader_election.cc:302] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91842a65d5344ed0b0b9ad2dc111f7a3, b2afa944a2a94d559ff1164bea12e594; no voters:
I20260620 14:32:15.869132 16041 raft_consensus.cc:2781] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:32:15.869208 16041 raft_consensus.cc:482] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:32:15.869246 16041 raft_consensus.cc:3037] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:15.870633 16041 raft_consensus.cc:504] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.871085 16041 leader_election.cc:288] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [CANDIDATE]: Term 1 election: Requested vote from peers b2afa944a2a94d559ff1164bea12e594 (127.4.58.65:43105), 1b22bb5f639f43c2819ba25eadbc50c0 (127.4.58.66:38111)
I20260620 14:32:15.871685 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c885267c854922bcd68e47c9fb75e2" candidate_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afa944a2a94d559ff1164bea12e594"
I20260620 14:32:15.871829 15700 raft_consensus.cc:3037] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:15.873219 15700 raft_consensus.cc:2454] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91842a65d5344ed0b0b9ad2dc111f7a3 in term 1.
I20260620 14:32:15.873704 15908 leader_election.cc:302] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91842a65d5344ed0b0b9ad2dc111f7a3, b2afa944a2a94d559ff1164bea12e594; no voters:
I20260620 14:32:15.873907 16041 raft_consensus.cc:2781] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:15.874132 16041 raft_consensus.cc:686] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 1 LEADER]: Becoming Leader. State: Replica: 91842a65d5344ed0b0b9ad2dc111f7a3, State: Running, Role: LEADER
I20260620 14:32:15.874548 16041 consensus_queue.cc:237] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } }
I20260620 14:32:15.878165 15534 catalog_manager.cc:5495] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91842a65d5344ed0b0b9ad2dc111f7a3 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41987 } health_report { overall_health: HEALTHY } } }
I20260620 14:32:15.879746 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.862385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:45081 (local address 127.4.58.66:38111)
0620 14:32:15.862515 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.862520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.863180 (+ 660us) server_negotiation.cc:407] Connection header received
0620 14:32:15.863380 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.863394 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.863446 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.863491 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:32:15.863900 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.864601 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.866097 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.866700 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.866780 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.866839 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.879491 (+ 12652us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.879615 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:32:15.881632 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.861717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38111 (local address 127.4.58.67:45081)
0620 14:32:15.863095 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.863109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:15.863249 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:15.863635 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:15.863638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:15.863655 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:32:15.863771 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.863777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.864768 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.864772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.865953 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:15.865963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.867183 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.867186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:15.867283 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.867307 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:32:15.880318 (+ 13011us) client_negotiation.cc:240] Negotiation successful
0620 14:32:15.881260 (+ 942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":15,"thread_start_us":115,"threads_started":1}
I20260620 14:32:15.882372 15838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c885267c854922bcd68e47c9fb75e2" candidate_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b22bb5f639f43c2819ba25eadbc50c0"
I20260620 14:32:15.882337 15839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c885267c854922bcd68e47c9fb75e2" candidate_uuid: "91842a65d5344ed0b0b9ad2dc111f7a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" is_pre_election: true
I20260620 14:32:15.882603 15838 raft_consensus.cc:3037] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:15.883834 15838 raft_consensus.cc:2454] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91842a65d5344ed0b0b9ad2dc111f7a3 in term 1.
I20260620 14:32:15.902019 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.896666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.4.58.67:41987)
0620 14:32:15.896834 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:15.896838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:15.896865 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:32:15.896962 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:15.896965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:15.897034 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:15.897155 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:32:15.897477 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.898251 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.899535 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:15.899840 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:15.899955 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:15.900456 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:32:15.901121 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:32:15.901241 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:15.901803 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 14:32:15.901873 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 14:32:15.905880 15886 tablet.cc:2369] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:32:15.910907 15838 raft_consensus.cc:1261] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 1 FOLLOWER]: Refusing update from remote peer 91842a65d5344ed0b0b9ad2dc111f7a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:15.911713 16050 consensus_queue.cc:1035] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b22bb5f639f43c2819ba25eadbc50c0" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:15.919584 15700 raft_consensus.cc:1261] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594 [term 1 FOLLOWER]: Refusing update from remote peer 91842a65d5344ed0b0b9ad2dc111f7a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:32:15.920426 16050 consensus_queue.cc:1035] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2afa944a2a94d559ff1164bea12e594" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:32:15.935009 16053 mvcc.cc:205] Tried to move back new op lower bound from 7298932473481129984 to 7298932473345216512. Current Snapshot: MvccSnapshot[applied={T|T < 7298932473481129984}]
I20260620 14:32:15.939915 16054 mvcc.cc:205] Tried to move back new op lower bound from 7298932473481129984 to 7298932473345216512. Current Snapshot: MvccSnapshot[applied={T|T < 7298932473481129984}]
I20260620 14:32:15.952735 15534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:15.952879 16056 mvcc.cc:205] Tried to move back new op lower bound from 7298932473481129984 to 7298932473345216512. Current Snapshot: MvccSnapshot[applied={T|T < 7298932473481129984 or (T in {7298932473481129984})}]
I20260620 14:32:15.952986 15534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:32:15.956128 15534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d49c8b928594c8e897726a05a625952: Sending DeleteTablet for 3 replicas of tablet f5c885267c854922bcd68e47c9fb75e2
I20260620 14:32:15.957087 15808 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c885267c854922bcd68e47c9fb75e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:15 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:51524
I20260620 14:32:15.957089 15683 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c885267c854922bcd68e47c9fb75e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:15 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:43042
I20260620 14:32:15.957644 16062 tablet_replica.cc:330] T f5c885267c854922bcd68e47c9fb75e2 P b2afa944a2a94d559ff1164bea12e594: stopping tablet replica
I20260620 14:32:15.958541 15953 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c885267c854922bcd68e47c9fb75e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:32:15 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:57894
I20260620 14:32:15.959141 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 15616
I20260620 14:32:15.967933 16063 tablet_replica.cc:330] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: stopping tablet replica
I20260620 14:32:15.968977 16064 tablet_replica.cc:330] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: stopping tablet replica
I20260620 14:32:15.969344 16064 raft_consensus.cc:2229] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:32:15.969396 16063 raft_consensus.cc:2229] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:32:15.969826 16064 raft_consensus.cc:2258] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:15.970357 16063 raft_consensus.cc:2258] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:32:15.972539 16064 ts_tablet_manager.cc:1905] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:32:15.975271 16063 ts_tablet_manager.cc:1905] T f5c885267c854922bcd68e47c9fb75e2 P 1b22bb5f639f43c2819ba25eadbc50c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:32:15.976081 16064 ts_tablet_manager.cc:1918] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:32:15.976258 16064 log.cc:1196] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/f5c885267c854922bcd68e47c9fb75e2
I20260620 14:32:15.976645 16064 ts_tablet_manager.cc:1939] T f5c885267c854922bcd68e47c9fb75e2 P 91842a65d5344ed0b0b9ad2dc111f7a3: Deleting consensus metadata
I20260620 14:32:15.978127 15520 catalog_manager.cc:4841] TS 91842a65d5344ed0b0b9ad2dc111f7a3 (127.4.58.67:41987): tablet f5c885267c854922bcd68e47c9fb75e2 (table test-table [id=e2d0b45a817b4b30b8ef67d0539288bf]) successfully deleted
W20260620 14:32:15.979030 15521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:43105 (error 108)
I20260620 14:32:15.979938 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 15755
I20260620 14:32:15.985828 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 15890
I20260620 14:32:15.986069 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.985736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:43105 (local address 127.0.0.1:48234)
0620 14:32:15.985914 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.985982 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 14:32:15.986929 15521 catalog_manager.cc:4584] TS b2afa944a2a94d559ff1164bea12e594 (127.4.58.65:43105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5c885267c854922bcd68e47c9fb75e2: Network error: Client connection negotiation failed: client connection to 127.4.58.65:43105: connect: Connection refused (error 111)
I20260620 14:32:15.987033 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:15.986752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:38111 (local address 127.0.0.1:45310)
0620 14:32:15.986900 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:15.986959 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:38111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:32:15.992806 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 15503
I20260620 14:32:16.001299 4329 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:32:16.337054 4329 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:32:16.337245 4329 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:32:16.672863 4329 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:32:16.673 UTC [15089] LOG: received smart shutdown request
2026-06-20 14:32:16.676 UTC [15089] LOG: background worker "logical replication launcher" (PID 15098) exited with exit code 1
2026-06-20 14:32:16.676 UTC [15093] LOG: shutting down
2026-06-20 14:32:16.856 UTC [15089] LOG: database system is shut down
2026-06-20T14:32:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46968 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:32:17 dist-test-slave-gqzk krb5kdc[16074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:32:17 dist-test-slave-gqzk krb5kdc[16074](info): set up 2 sockets
Jun 20 14:32:17 dist-test-slave-gqzk krb5kdc[16074](info): commencing operation
krb5kdc: starting...
W20260620 14:32:17.912698 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:32:17 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:32:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangeradmin_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangeradmin_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangerlookup_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangerlookup_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
I20260620 14:32:18.084353 4329 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCj3QH_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/postgres -l logfile start
2026-06-20 14:32:20.539 UTC [16104] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:32:20.539 UTC [16104] LOG: listening on IPv4 address "127.4.58.84", port 50339
2026-06-20 14:32:20.543 UTC [16104] LOG: listening on Unix socket "/tmp/.s.PGSQL.50339"
2026-06-20 14:32:20.571 UTC [16107] LOG: database system was shut down at 2026-06-20 14:32:19 UTC
2026-06-20 14:32:20.575 UTC [16104] LOG: database system is ready to accept connections
I20260620 14:32:21.444628 4329 mini_postgres.cc:96] Postgres bound to 50339
2026-06-20 14:32:21.450 UTC [16115] FATAL: database "slave" does not exist
127.4.58.84:50339 - accepting connections
I20260620 14:32:21.451313 4329 mini_ranger.cc:161] Starting Ranger...
I20260620 14:32:21.466440 4329 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:32:21.934508 4329 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:32:21.934589 4329 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin
2026-06-20 14:32:22,195 [I] DB FLAVOR :POSTGRES
2026-06-20 14:32:22,195 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:32:22,195 [I] Checking connection..
2026-06-20 14:32:22,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:32:22,455 [I] Checking connection passed.
2026-06-20 14:32:22,455 [I] --------- Verifying version history table ---------
2026-06-20 14:32:22,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:22,700 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:32:22,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:22,933 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:32:22,933 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:32:22,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:32:23.159 UTC [16179] WARNING: there is no transaction in progress
2026-06-20 14:32:23,164 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:32:23,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:32:23,412 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:32:23,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:32:23,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:32:23,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:32:23,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gqzk.c.gcp-upstream.internal','N') ;"
2026-06-20 14:32:24,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:32:24,360 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:32:24,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:32:24,605 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:32:24,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:32:24,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:32:25,115 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:32:25,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:32:26.123 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.139 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.156 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.174 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.191 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.210 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.266 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.273 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.290 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.301 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.312 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.327 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.337 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.344 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.351 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:26.357 UTC [16287] WARNING: there is no transaction in progress
2026-06-20 14:32:27,038 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:32:27,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:32:27,267 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:32:27,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:32:27,507 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:32:27,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:32:27,765 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:32:27,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:32:27,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:32:28,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gqzk.c.gcp-upstream.internal';"
2026-06-20 14:32:28,498 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:32:28,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:32:28,740 [I] DB_PATCHES have already been applied
I20260620 14:32:28.746088 4329 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCj3QH_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:32:28.746150 4329 mini_ranger.cc:193] Using host: 127.4.58.84
I20260620 14:32:28.749446 4329 mini_ranger.cc:227] Ranger admin URL: http://127.4.58.84:33433
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:32:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:32:29.738813 4329 mini_ranger.cc:160] Time spent starting Ranger: real 8.288s user 0.005s sys 0.004s
Jun 20, 2026 2:32:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:32:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:32:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:32:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:32:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:32:50.078249 4329 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.58.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.58.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangerkms_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.58.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/rangerkms_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.58.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.58.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.58.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/HTTP_127.4.58.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:32:50 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965970, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:32:50.175994 4329 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:32:50.186398 4329 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:32:51.038443 4329 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:32:51.038524 4329 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms
I20260620 14:32:51.038542 4329 mini_ranger_kms.cc:227] Using postgres at 127.4.58.84:50339
2026-06-20 14:32:51,194 [I] DB FLAVOR :POSTGRES
2026-06-20 14:32:51,194 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:32:51,194 [I] Checking connection
2026-06-20 14:32:51,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:32:51,434 [I] connection success
2026-06-20 14:32:51,434 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:32:51,434 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:32:51,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:32:51,695 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:32:51,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:32:51,695 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:32:51,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.58.84:50339/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:32:51,947 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:32:52.382683 4329 mini_ranger_kms.cc:310] Created kms service
I20260620 14:32:52.583206 4329 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:32:52.630479 4329 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:32:52 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965970, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:53.038722 4329 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:32:53.038872 4329 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCj3QH_/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCj3QH_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:32:53.038892 4329 mini_ranger_kms.cc:240] Using host: 127.4.58.84
I20260620 14:32:53.042376 4329 mini_ranger_kms.cc:276] Ranger KMS PID: 16467
I20260620 14:32:53.042462 4329 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.58.84:52001
Jun 20, 2026 2:32:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:32:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:32:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:32:55.008430 4329 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.832s user 0.007s sys 0.132s
I20260620 14:32:55.008548 4329 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:32:55.008621 4329 mini_ranger_kms.cc:401] 127.4.58.84:52001/kms/v1/keys
Jun 20 14:32:55 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965975, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:32:55 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965975, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
Jun 20 14:32:55 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965975, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.58.84@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
Jun 20 14:32:58 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:32:58.315178 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44067
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44067
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:58.448618 16518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:58.448966 16518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:58.449009 16518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:58.452940 16518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:32:58.452992 16518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:58.453009 16518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:58.453024 16518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:32:58.453037 16518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:32:58.458417 16518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.58.84:52001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44067
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:58.459412 16518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:58.460372 16518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:58.468986 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:58.469112 16518 server_base.cc:1034] running on GCE node
W20260620 14:32:58.468990 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:58.469058 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:58.469810 16518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:58.470922 16518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:58.472090 16518 hybrid_clock.cc:648] HybridClock initialized: now 1781965978472060 us; error 44 us; skew 500 ppm
Jun 20 14:32:58 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:58.478232 16518 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:32:58.479146 16518 webserver.cc:460] Webserver started at http://127.4.58.126:38291/ using document root <none> and password file <none>
I20260620 14:32:58.479658 16518 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:58.479708 16518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:58.479872 16518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:58 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:58.529064 16518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "beef2bb883bd428e832f4c4404d1fa95"
format_stamp: "Formatted at 2026-06-20 14:32:58 on dist-test-slave-gqzk"
server_key: "d979d0076e2ded3341868440fa533b29"
server_key_iv: "9521c8bcb72ec85c990c4e50d8184f3e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:58.529824 16518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "beef2bb883bd428e832f4c4404d1fa95"
format_stamp: "Formatted at 2026-06-20 14:32:58 on dist-test-slave-gqzk"
server_key: "d979d0076e2ded3341868440fa533b29"
server_key_iv: "9521c8bcb72ec85c990c4e50d8184f3e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:58.535161 16518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:32:58.550969 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:58.552366 16518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:32:58.552486 16518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "beef2bb883bd428e832f4c4404d1fa95"
format_stamp: "Formatted at 2026-06-20 14:32:58 on dist-test-slave-gqzk"
server_key: "d979d0076e2ded3341868440fa533b29"
server_key_iv: "9521c8bcb72ec85c990c4e50d8184f3e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:58.552649 16518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:58.580544 16518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:58.583706 16518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:58.583881 16518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:58.591764 16518 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44067
I20260620 14:32:58.591781 16585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44067 every 8 connection(s)
I20260620 14:32:58.592916 16518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:32:58.593411 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 16518
I20260620 14:32:58.593542 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:32:58.593816 4329 external_mini_cluster.cc:1428] Setting key f353fa2d4407c7196bacae6ad0791103
I20260620 14:32:58.596580 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:32:58 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:32:58.603960 16586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Bootstrap starting.
I20260620 14:32:58.606770 16586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Neither blocks nor log segments found. Creating new log.
I20260620 14:32:58.607615 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:58.595387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.4.58.126:44067)
0620 14:32:58.595829 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:58.595845 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:58.595883 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:32:58.596537 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:58.596555 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:58.596984 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:58.597382 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:58.598243 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.599299 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.600195 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:58.600678 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:58.600851 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:58.603257 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:58.603263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:58.603277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:58.603306 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:58.604836 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:58.605256 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:58.605262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:58.605266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:58.605385 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:58.605643 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:58.605646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:58.605647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:58.606047 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:58.606232 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:58.606525 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:32:58.606813 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":111,"threads_started":1}
I20260620 14:32:58.608109 16586 log.cc:824] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Log is configured to *not* fsync() on all Append() calls
I20260620 14:32:58.610146 16586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: No bootstrap required, opened a new log
I20260620 14:32:58.613159 16586 raft_consensus.cc:348] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:32:58.613400 16586 raft_consensus.cc:374] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:32:58.613456 16586 raft_consensus.cc:729] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beef2bb883bd428e832f4c4404d1fa95, State: Initialized, Role: FOLLOWER
I20260620 14:32:58.613914 16586 consensus_queue.cc:260] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:32:58.614058 16586 raft_consensus.cc:388] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:32:58.614121 16586 raft_consensus.cc:482] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:32:58.614220 16586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:32:58.615144 16586 raft_consensus.cc:504] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:32:58.615489 16586 leader_election.cc:302] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beef2bb883bd428e832f4c4404d1fa95; no voters:
I20260620 14:32:58.615854 16586 leader_election.cc:288] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:32:58.615939 16591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:32:58.616153 16591 raft_consensus.cc:686] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 LEADER]: Becoming Leader. State: Replica: beef2bb883bd428e832f4c4404d1fa95, State: Running, Role: LEADER
I20260620 14:32:58.616513 16591 consensus_queue.cc:237] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:32:58.617274 16586 sys_catalog.cc:564] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:32:58.617789 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beef2bb883bd428e832f4c4404d1fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } } }
I20260620 14:32:58.617981 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:58.618242 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beef2bb883bd428e832f4c4404d1fa95. Latest consensus state: current_term: 1 leader_uuid: "beef2bb883bd428e832f4c4404d1fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } } }
I20260620 14:32:58.618376 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:32:58.618678 16598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:32:58.622074 16598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:32:58.622489 16586 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:32:58.640215 16598 catalog_manager.cc:1347] Generated new cluster ID: 9ccbb836110b4b2386ed4cd8e6e84d6b
I20260620 14:32:58.640407 16598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:32:58.665710 16598 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:32:58.667265 16598 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:32:58.679395 16598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Generated new TSK 0
I20260620 14:32:58.680434 16598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:32:59.356145 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:32:59.477880 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:59.478175 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:59.478238 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:59.482594 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:59.482661 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:59.482740 16631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:32:59.488257 16631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:59.489526 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:59.490649 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:59.500937 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:59.501108 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:59.500962 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:59.501791 16631 server_base.cc:1034] running on GCE node
I20260620 14:32:59.502338 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:59.503224 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:59.504431 16631 hybrid_clock.cc:648] HybridClock initialized: now 1781965979504401 us; error 56 us; skew 500 ppm
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:59.508132 16631 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:32:59.509383 16631 webserver.cc:460] Webserver started at http://127.4.58.65:41883/ using document root <none> and password file <none>
I20260620 14:32:59.510134 16631 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:59.510236 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:59.510488 16631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:59.528244 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "849f5723f3a441d18b8226900e444166"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "76af748035185ec5e76a0d383ebbba2c"
server_key_iv: "3c188084023083dde944c65f536d932e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.529100 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "849f5723f3a441d18b8226900e444166"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "76af748035185ec5e76a0d383ebbba2c"
server_key_iv: "3c188084023083dde944c65f536d932e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.533663 16631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:32:59.556936 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:59.558792 16631 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:32:59.559052 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "849f5723f3a441d18b8226900e444166"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "76af748035185ec5e76a0d383ebbba2c"
server_key_iv: "3c188084023083dde944c65f536d932e"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.559307 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:59.582170 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:59.586205 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:59.586493 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:59.587167 16631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:59.588186 16631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:59.588238 16631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:59.588291 16631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:59.588333 16631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:59.598636 16631 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:41901
I20260620 14:32:59.599826 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:32:59.606637 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 16631
I20260620 14:32:59.606784 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:32:59.607131 4329 external_mini_cluster.cc:1428] Setting key 5c855eaa1f3274efcd40271214919006
I20260620 14:32:59.616117 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:41901 every 8 connection(s)
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:32:59.639256 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:59.615847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55283 (local address 127.4.58.126:44067)
0620 14:32:59.616906 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0620 14:32:59.616916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:32:59.618523 (+ 1607us) server_negotiation.cc:407] Connection header received
0620 14:32:59.618587 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:32:59.618590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:32:59.618650 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:32:59.618739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:32:59.620342 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:59.621219 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:59.624861 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:32:59.625244 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:32:59.625338 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:59.629871 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:32:59.629875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:32:59.629878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:32:59.629911 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:32:59.631747 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:59.632438 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:59.632442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:59.632444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:59.632540 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:32:59.633021 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:32:59.633025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:32:59.633027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:32:59.633266 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:32:59.633352 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:32:59.638898 (+ 5546us) server_negotiation.cc:299] Negotiation successful
0620 14:32:59.639064 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":100,"threads_started":1}
I20260620 14:32:59.641331 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:32:59.614590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.65:55283)
0620 14:32:59.617035 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0620 14:32:59.617095 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:32:59.618206 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:32:59.619089 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:32:59.619099 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:32:59.619468 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:32:59.620169 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:59.620189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:59.621367 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:59.621371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:59.622944 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:32:59.622955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:32:59.625589 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:32:59.625593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:32:59.625818 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:32:59.626542 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:32:59.626565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:32:59.628306 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:32:59.631906 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:59.631914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:59.631935 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:59.632321 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:59.632663 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:32:59.632666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:32:59.632668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:32:59.632911 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:32:59.633429 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:32:59.633439 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:32:59.633917 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 14:32:59.639751 (+ 5834us) client_negotiation.cc:240] Negotiation successful
0620 14:32:59.640164 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"thread_start_us":195,"threads_started":1}
I20260620 14:32:59.643162 16762 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:32:59.643420 16762 heartbeater.cc:464] Registering TS with master...
I20260620 14:32:59.644049 16762 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:32:59.646072 16550 ts_manager.cc:194] Registered new tserver with Master: 849f5723f3a441d18b8226900e444166 (127.4.58.65:41901)
I20260620 14:32:59.647694 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:55283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:32:59.688275 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:32:59.862001 16770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:32:59.862404 16770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:32:59.862479 16770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:32:59.868548 16770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:32:59.868642 16770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:32:59.868732 16770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:32:59.876821 16770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:32:59.878348 16770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:32:59.879769 16770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:32:59.893124 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:59.893601 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:32:59.894178 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:32:59.895117 16770 server_base.cc:1034] running on GCE node
I20260620 14:32:59.895720 16770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:32:59.896608 16770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:32:59.897820 16770 hybrid_clock.cc:648] HybridClock initialized: now 1781965979897756 us; error 92 us; skew 500 ppm
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:32:59.901877 16770 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:32:59.903142 16770 webserver.cc:460] Webserver started at http://127.4.58.66:37383/ using document root <none> and password file <none>
I20260620 14:32:59.903950 16770 fs_manager.cc:359] Metadata directory not provided
I20260620 14:32:59.904105 16770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:32:59.905023 16770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:32:59 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:32:59.928826 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "a89c63e947c2446f8398694f33bde748"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "49125df0da08a7a928576d4d7cbf9993"
server_key_iv: "9010cd958cb2b32ccfafb043468f94a0"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.929670 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "a89c63e947c2446f8398694f33bde748"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "49125df0da08a7a928576d4d7cbf9993"
server_key_iv: "9010cd958cb2b32ccfafb043468f94a0"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.935201 16770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:32:59.953615 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:59.955361 16770 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 14:32:59.955508 16770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "a89c63e947c2446f8398694f33bde748"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "49125df0da08a7a928576d4d7cbf9993"
server_key_iv: "9010cd958cb2b32ccfafb043468f94a0"
server_key_version: "kuduclusterkey@0"
I20260620 14:32:59.955672 16770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:32:59.982149 16770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:32:59.985922 16770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:32:59.986151 16770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:32:59.986877 16770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:32:59.987954 16770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:32:59.988026 16770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:32:59.988096 16770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:32:59.988125 16770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.005118 16770 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:34957
I20260620 14:33:00.006963 16770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:00.007951 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:34957 every 8 connection(s)
I20260620 14:33:00.010767 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 16770
I20260620 14:33:00.010918 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:33:00.011242 4329 external_mini_cluster.cc:1428] Setting key 633877daf0228d83027d47675695b3b9
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:00.056876 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:00.013185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:42827 (local address 127.4.58.126:44067)
0620 14:33:00.013354 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:00.013359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:00.016393 (+ 3034us) server_negotiation.cc:407] Connection header received
0620 14:33:00.026394 (+ 10001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:00.026398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:00.026479 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:00.026700 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:00.037098 (+ 10398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.037947 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.041659 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.042029 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.042115 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:00.050965 (+ 8850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:00.050969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:00.050972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:00.051010 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:00.052785 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:00.054501 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:00.054505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:00.054507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:00.054614 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:00.055116 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:00.055119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:00.055121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:00.055552 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:00.055643 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:00.056487 (+ 844us) server_negotiation.cc:299] Negotiation successful
0620 14:33:00.056639 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:33:00.058688 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:00.011474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.66:42827)
0620 14:33:00.016142 (+ 4668us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:00.016249 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:00.026050 (+ 9801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:00.028476 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:00.028489 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:00.035831 (+ 7342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:00.036810 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:00.036853 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.038169 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.038172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:00.039685 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:00.039700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.042369 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.042373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:00.042630 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:00.043744 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:00.043767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:00.050634 (+ 6867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:00.052969 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:00.052977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:00.053000 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:00.053416 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:00.054734 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:00.054738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:00.054740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:00.054994 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:00.055715 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:00.055722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:00.056177 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 14:33:00.057079 (+ 902us) client_negotiation.cc:240] Negotiation successful
0620 14:33:00.057499 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4308,"thread_start_us":318,"threads_started":1}
I20260620 14:33:00.060180 16900 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:33:00.060470 16900 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:00.061168 16900 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:00.062731 16550 ts_manager.cc:194] Registered new tserver with Master: a89c63e947c2446f8398694f33bde748 (127.4.58.66:34957)
I20260620 14:33:00.063493 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:42827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:33:00.071712 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:00.224347 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:00.224648 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:00.224709 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:00.229101 16905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:00.229178 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:00.229259 16905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:00.235234 16905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:00.236362 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:00.237488 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:00.247359 16910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.247357 16913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.247807 16911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:00.247799 16905 server_base.cc:1034] running on GCE node
I20260620 14:33:00.248327 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:00.249179 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:00.250355 16905 hybrid_clock.cc:648] HybridClock initialized: now 1781965980250277 us; error 92 us; skew 500 ppm
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:00.253605 16905 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:00.254563 16905 webserver.cc:460] Webserver started at http://127.4.58.67:41339/ using document root <none> and password file <none>
I20260620 14:33:00.255111 16905 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:00.255177 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:00.255378 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:00.276737 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "b2a6e8deaaa0478882f461fdb211224a"
format_stamp: "Formatted at 2026-06-20 14:33:00 on dist-test-slave-gqzk"
server_key: "f40ebf515750edc66ff901762d9af26c"
server_key_iv: "d13094eee0fe23392c6cf29591e108d4"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:00.278296 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "b2a6e8deaaa0478882f461fdb211224a"
format_stamp: "Formatted at 2026-06-20 14:33:00 on dist-test-slave-gqzk"
server_key: "f40ebf515750edc66ff901762d9af26c"
server_key_iv: "d13094eee0fe23392c6cf29591e108d4"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:00.286864 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965979, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:00.312929 16920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.314915 16905 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260620 14:33:00.315163 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "b2a6e8deaaa0478882f461fdb211224a"
format_stamp: "Formatted at 2026-06-20 14:33:00 on dist-test-slave-gqzk"
server_key: "f40ebf515750edc66ff901762d9af26c"
server_key_iv: "d13094eee0fe23392c6cf29591e108d4"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:00.315443 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:00.348206 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:00.352524 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:00.352818 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:00.353889 16905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:00.355087 16905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:00.355212 16905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.355319 16905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:00.355386 16905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.369877 16905 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:41995
I20260620 14:33:00.371315 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:00.373201 17033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:41995 every 8 connection(s)
I20260620 14:33:00.375278 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 16905
I20260620 14:33:00.375403 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:33:00.375936 4329 external_mini_cluster.cc:1428] Setting key de24957b7d7ac7ec45d32b5c07b0d846
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:00.413450 17036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:00.379459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.67:49529)
0620 14:33:00.385143 (+ 5684us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:00.385220 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:00.386472 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:00.387453 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:00.387465 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:00.387913 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:00.388678 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:00.388699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.402094 (+ 13395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.402099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:00.403760 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:00.403779 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.404681 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.404686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:00.405019 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:00.405807 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:00.405830 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:00.407748 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:00.409846 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:00.409853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:00.409874 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:00.410251 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:00.410615 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:00.410619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:00.410620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:00.410812 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:00.411310 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:00.411317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:00.411682 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 14:33:00.411893 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 14:33:00.412269 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5387,"thread_start_us":182,"threads_started":1}
I20260620 14:33:00.414182 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:00.380693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:49529 (local address 127.4.58.126:44067)
0620 14:33:00.380854 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:00.380859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:00.385335 (+ 4476us) server_negotiation.cc:407] Connection header received
0620 14:33:00.386768 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:00.386773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:00.386836 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:00.386916 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:00.400964 (+ 14048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.401832 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.403958 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:00.404331 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:00.404426 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:00.407985 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:00.407990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:00.407992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:00.408024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:00.409683 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:00.410376 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:00.410380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:00.410382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:00.410490 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:00.410918 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:00.410921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:00.410923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:00.411168 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:00.411256 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:00.413873 (+ 2617us) server_negotiation.cc:299] Negotiation successful
0620 14:33:00.414022 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:33:00.415755 17034 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:33:00.416011 17034 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:00.416594 17034 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:00.417867 16550 ts_manager.cc:194] Registered new tserver with Master: b2a6e8deaaa0478882f461fdb211224a (127.4.58.67:41995)
I20260620 14:33:00.418648 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:49529
I20260620 14:33:00.426030 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:00.426882 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 16631
I20260620 14:33:00.436926 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 16770
I20260620 14:33:00.445075 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 16905
I20260620 14:33:00.451633 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 16518
I20260620 14:33:00.458235 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44067
--webserver_interface=127.4.58.126
--webserver_port=38291
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44067
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:00.570716 17037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:00.571014 17037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:00.571071 17037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:00.575325 17037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:00.575400 17037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:00.575426 17037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:00.575449 17037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:00.575479 17037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:00.581314 17037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44067
--ranger_config_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=38291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:00.582527 17037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:00.583647 17037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:00.592649 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.592707 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.592615 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:00.593103 17037 server_base.cc:1034] running on GCE node
I20260620 14:33:00.593778 17037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:00.595218 17037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:00.596398 17037 hybrid_clock.cc:648] HybridClock initialized: now 1781965980596366 us; error 45 us; skew 500 ppm
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:00.599733 17037 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:00.600888 17037 webserver.cc:460] Webserver started at http://127.4.58.126:38291/ using document root <none> and password file <none>
I20260620 14:33:00.601553 17037 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:00.601640 17037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:00.685128 17037 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:33:00.688091 17053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.689759 17037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:00.689908 17037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "beef2bb883bd428e832f4c4404d1fa95"
format_stamp: "Formatted at 2026-06-20 14:32:58 on dist-test-slave-gqzk"
server_key: "d979d0076e2ded3341868440fa533b29"
server_key_iv: "9521c8bcb72ec85c990c4e50d8184f3e"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:00.690661 17037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:00.718485 17037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:00.721351 17037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:00.721509 17037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:00.728582 17037 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44067
I20260620 14:33:00.728632 17105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44067 every 8 connection(s)
I20260620 14:33:00.729650 17037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:00.733942 17106 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:00.735606 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Bootstrap starting.
I20260620 14:33:00.736454 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17037
I20260620 14:33:00.736994 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:41901
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:44067
--webserver_port=41883
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:00.752624 17106 log.cc:824] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:00.759687 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:00.760080 17106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Bootstrap complete.
I20260620 14:33:00.764147 17106 raft_consensus.cc:348] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:33:00.764564 17106 raft_consensus.cc:729] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: beef2bb883bd428e832f4c4404d1fa95, State: Initialized, Role: FOLLOWER
I20260620 14:33:00.765131 17106 consensus_queue.cc:260] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:33:00.765251 17106 raft_consensus.cc:388] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:00.765316 17106 raft_consensus.cc:482] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:00.765404 17106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:33:00.767351 17106 raft_consensus.cc:504] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:33:00.767678 17106 leader_election.cc:302] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beef2bb883bd428e832f4c4404d1fa95; no voters:
I20260620 14:33:00.767987 17106 leader_election.cc:288] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:33:00.768106 17110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:33:00.768352 17110 raft_consensus.cc:686] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [term 2 LEADER]: Becoming Leader. State: Replica: beef2bb883bd428e832f4c4404d1fa95, State: Running, Role: LEADER
I20260620 14:33:00.768683 17110 consensus_queue.cc:237] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } }
I20260620 14:33:00.769246 17106 sys_catalog.cc:564] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:00.770114 17111 sys_catalog.cc:455] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "beef2bb883bd428e832f4c4404d1fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } } }
I20260620 14:33:00.770239 17111 sys_catalog.cc:458] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:00.770509 17112 sys_catalog.cc:455] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beef2bb883bd428e832f4c4404d1fa95. Latest consensus state: current_term: 2 leader_uuid: "beef2bb883bd428e832f4c4404d1fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beef2bb883bd428e832f4c4404d1fa95" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44067 } } }
I20260620 14:33:00.770586 17112 sys_catalog.cc:458] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:00.771306 17119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:00.771422 17106 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:33:00.777360 17119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:00.778126 17119 catalog_manager.cc:1259] Loaded cluster ID: 9ccbb836110b4b2386ed4cd8e6e84d6b
I20260620 14:33:00.778192 17119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:00.780679 17119 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:00.781558 17119 catalog_manager.cc:5879] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Loaded TSK: 0
I20260620 14:33:00.782856 17119 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:33:00.888571 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:00.889010 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:00.889098 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:00.895364 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:00.895464 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:00.895560 17108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:00.904151 17108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:41901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=41883
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:00.905640 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:00.907047 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:00.917358 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.917501 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:00.917358 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:00.917733 17108 server_base.cc:1034] running on GCE node
I20260620 14:33:00.918746 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:00.919466 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:00.920709 17108 hybrid_clock.cc:648] HybridClock initialized: now 1781965980920671 us; error 58 us; skew 500 ppm
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:00.924410 17108 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:00.925486 17108 webserver.cc:460] Webserver started at http://127.4.58.65:41883/ using document root <none> and password file <none>
I20260620 14:33:00.926234 17108 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:00.926337 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:00 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:00.951711 17108 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:00.954558 17154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.956125 17108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:33:00.956259 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "849f5723f3a441d18b8226900e444166"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "76af748035185ec5e76a0d383ebbba2c"
server_key_iv: "3c188084023083dde944c65f536d932e"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:00.956950 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:00.983872 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:00.991927 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:00.992192 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:00.992861 17108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:00.993790 17108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:00.993829 17108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:00.993868 17108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:00.993882 17108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.004278 17108 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:41901
I20260620 14:33:01.004596 17267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:41901 every 8 connection(s)
I20260620 14:33:01.005523 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:01.014779 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17108
I20260620 14:33:01.015261 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:34957
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:44067
--webserver_port=37383
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:01.026827 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.008211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.65:51469)
0620 14:33:01.009175 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:01.009241 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:01.010175 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:01.011788 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:01.011799 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:01.012199 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:01.012959 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.012997 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.014533 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.014538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.016056 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.016069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.017412 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.017417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.017646 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.018358 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:01.018381 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:01.020250 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:01.022957 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.022965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.022982 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.023363 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.023816 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.023820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.023822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.024008 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.024824 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:01.024831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:01.025174 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 14:33:01.025372 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:33:01.025744 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":137,"threads_started":1}
I20260620 14:33:01.028867 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.008038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:51469 (local address 127.4.58.126:44067)
0620 14:33:01.010646 (+ 2608us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:01.010665 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:01.010706 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:33:01.010948 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:01.010960 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:01.011323 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:01.011620 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:01.013195 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.014365 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.016199 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.016975 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.017156 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.020537 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:01.020558 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:01.020579 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:01.020618 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:01.022792 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.023516 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.023524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.023529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.023704 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.024118 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.024121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.024123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.024661 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:01.027360 (+ 2699us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:01.027538 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:33:01.027861 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":146,"threads_started":1}
I20260620 14:33:01.032325 17268 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:33:01.032609 17268 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.033380 17268 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
W20260620 14:33:01.035827 17268 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:44067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:33:01.035960 17268 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.036147 17268 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:01.037007 17268 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.037200 17268 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
W20260620 14:33:01.039075 17268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:01.166908 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:01.167181 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:01.167222 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:01.171746 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:01.171826 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:01.171907 17272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:01.178519 17272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:34957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=37383
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:01.180066 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:01.181414 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:01.194170 17278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:01.194463 17272 server_base.cc:1034] running on GCE node
W20260620 14:33:01.194134 17277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:01.194226 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:01.195218 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:01.196082 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:01.197387 17272 hybrid_clock.cc:648] HybridClock initialized: now 1781965981197308 us; error 89 us; skew 500 ppm
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:01.201069 17272 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:01.202256 17272 webserver.cc:460] Webserver started at http://127.4.58.66:37383/ using document root <none> and password file <none>
I20260620 14:33:01.202991 17272 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:01.203099 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:01.227219 17272 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:01.230983 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.232668 17272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:01.232869 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "a89c63e947c2446f8398694f33bde748"
format_stamp: "Formatted at 2026-06-20 14:32:59 on dist-test-slave-gqzk"
server_key: "49125df0da08a7a928576d4d7cbf9993"
server_key_iv: "9010cd958cb2b32ccfafb043468f94a0"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:01.233641 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:01.258409 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:01.262380 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:01.262676 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:01.263397 17272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:01.264619 17272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:01.264688 17272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.264775 17272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:01.264809 17272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 14:33:01.275919 17272 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:34957
I20260620 14:33:01.277297 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:01.278257 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:34957 every 8 connection(s)
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:01.286893 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17272
I20260620 14:33:01.287406 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.58.84:52001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:41995
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:44067
--webserver_port=41339
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:01.291638 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.279330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:58753 (local address 127.4.58.126:44067)
0620 14:33:01.279490 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:01.279495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:01.280412 (+ 917us) server_negotiation.cc:407] Connection header received
0620 14:33:01.281533 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:01.281537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:01.281592 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:01.281667 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:01.282777 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.283294 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.284447 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.284669 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.284780 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.287440 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:01.287443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:01.287445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:01.287501 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:01.288992 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.289686 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.289690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.289692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.289794 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.290227 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.290229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.290231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.290449 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:01.290572 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:01.291305 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 14:33:01.291470 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:33:01.293372 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.279747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.66:58753)
0620 14:33:01.280212 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:01.280303 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:01.281319 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:01.281864 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:01.281871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:01.282143 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:01.282633 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.282647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.283411 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.283415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.284325 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.284333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.284839 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.284842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.285080 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.285715 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:01.285733 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:01.287257 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:01.289134 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.289142 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.289163 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.289573 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.289904 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.289907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.289909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.290122 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.290630 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:01.290638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:01.291001 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 14:33:01.291892 (+ 891us) client_negotiation.cc:240] Negotiation successful
0620 14:33:01.292253 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260620 14:33:01.294847 17403 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:33:01.295116 17403 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.295787 17403 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
W20260620 14:33:01.297603 17403 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:44067 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:33:01.297756 17403 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.297942 17403 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:01.298799 17403 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.298982 17403 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
W20260620 14:33:01.299721 17403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:01.431802 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:01.432169 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:01.432348 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:01.438210 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:01.438278 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:01.438359 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:01.444197 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.58.84:52001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:41995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=41339
--tserver_master_addrs=127.4.58.126:44067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:01.445356 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:01.446470 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:01.457177 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:01.457228 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:01.457398 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:01.458674 17406 server_base.cc:1034] running on GCE node
I20260620 14:33:01.459192 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:01.460008 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:01.461185 17406 hybrid_clock.cc:648] HybridClock initialized: now 1781965981461148 us; error 52 us; skew 500 ppm
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:01.465523 17406 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:01.466748 17406 webserver.cc:460] Webserver started at http://127.4.58.67:41339/ using document root <none> and password file <none>
I20260620 14:33:01.467478 17406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:01.467602 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:01.485368 17406 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:01.488422 17422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.490295 17406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:01.490447 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "b2a6e8deaaa0478882f461fdb211224a"
format_stamp: "Formatted at 2026-06-20 14:33:00 on dist-test-slave-gqzk"
server_key: "f40ebf515750edc66ff901762d9af26c"
server_key_iv: "d13094eee0fe23392c6cf29591e108d4"
server_key_version: "kuduclusterkey@0"
I20260620 14:33:01.491220 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:01.525064 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:01.528733 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:01.529017 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:01.529682 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:01.530591 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:01.530640 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.530679 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:01.530709 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:01.541065 17406 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:41995
I20260620 14:33:01.542230 17535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:41995 every 8 connection(s)
I20260620 14:33:01.542619 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:01.545984 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17406
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:01.559353 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.547127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.4.58.126:44067)
0620 14:33:01.549701 (+ 2574us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:01.549705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:01.549716 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:33:01.549745 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:01.549748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:01.549782 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:01.549855 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:01.550874 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.551717 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.553051 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.553420 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.553534 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.555391 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:01.555395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:01.555398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:01.555427 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:01.557497 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.557995 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.557998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.558000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.558086 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.558398 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.558402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.558404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.558630 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:01.558812 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:01.559114 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:33:01.559219 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2504,"thread_start_us":96,"threads_started":1}
I20260620 14:33:01.564311 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.545651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:44475 (local address 127.4.58.126:44067)
0620 14:33:01.545805 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:01.545809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:01.548295 (+ 2486us) server_negotiation.cc:407] Connection header received
0620 14:33:01.549515 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:01.549518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:01.549577 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:01.549654 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:01.553225 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.554012 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.555779 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.556227 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.556299 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.560532 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:01.560536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:01.560538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:01.560562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:01.562249 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.562781 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.562785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.562787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.562872 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:01.563197 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:01.563200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:01.563202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:01.563408 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:01.563510 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:01.564039 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:33:01.564162 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:33:01.565382 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:01.546413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:44067 (local address 127.4.58.67:44475)
0620 14:33:01.548130 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:01.548192 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:01.549287 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:01.550008 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:01.550018 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:01.550371 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:01.553012 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.553039 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.554179 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.554182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.555626 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:01.555639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:01.556529 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:01.556533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:01.556805 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:01.557382 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:01.557403 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:01.560178 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:01.562374 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.562380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.562395 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.562675 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.562956 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:01.562958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:01.562960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:01.563110 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:01.563502 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:01.563507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:01.563776 (+ 269us) client_negotiation.cc:769] Sending connection context
0620 14:33:01.563932 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:33:01.564237 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1472,"thread_start_us":149,"threads_started":1}
I20260620 14:33:01.566694 17536 heartbeater.cc:347] Connected to a master server at 127.4.58.126:44067
I20260620 14:33:01.566953 17536 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:01.567605 17536 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:01.569038 17070 ts_manager.cc:194] Registered new tserver with Master: b2a6e8deaaa0478882f461fdb211224a (127.4.58.67:41995)
I20260620 14:33:01.570297 17070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:44475
Jun 20 14:33:01 dist-test-slave-gqzk krb5kdc[16074](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:02.039322 17268 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:02.039640 17268 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:02.040467 17070 ts_manager.cc:194] Registered new tserver with Master: 849f5723f3a441d18b8226900e444166 (127.4.58.65:41901)
I20260620 14:33:02.041415 17070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:51469
Jun 20 14:33:02 dist-test-slave-gqzk krb5kdc[16074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for HTTP/127.4.58.84@KRBTEST.COM
I20260620 14:33:02.300051 17403 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:02.300369 17403 heartbeater.cc:511] Master 127.4.58.126:44067 requested a full tablet report, sending...
I20260620 14:33:02.301352 17070 ts_manager.cc:194] Registered new tserver with Master: a89c63e947c2446f8398694f33bde748 (127.4.58.66:34957)
I20260620 14:33:02.302136 17070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:58753
I20260620 14:33:02.307399 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:02.319922 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.309303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.4.58.126:44067)
0620 14:33:02.309631 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.309635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.310027 (+ 392us) server_negotiation.cc:407] Connection header received
0620 14:33:02.310169 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.310173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.310235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.310333 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:02.311170 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.311968 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.313184 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.313561 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.313635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.315385 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:02.315389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:02.315392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:02.315423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:02.317391 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:02.317978 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:02.317983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:02.317984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:02.318086 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:02.318823 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:02.318828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:02.318830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:02.319131 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:02.319290 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.319552 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.319711 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260620 14:33:02.324704 17070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:02.333320 17070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:02.358721 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.349796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:34957 (local address 127.0.0.1:35702)
0620 14:33:02.350279 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.350338 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:02.350475 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:02.353268 (+ 2793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:02.353271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:02.353290 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:02.353389 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.353395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.355247 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.355250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.356991 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.357001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.358158 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.358162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.358280 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.358327 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:33:02.358415 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:33:02.358538 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":231,"thread_start_us":160,"threads_started":1}
I20260620 14:33:02.359776 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.351180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.4.58.66:34957)
0620 14:33:02.352800 (+ 1620us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.352807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.352829 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:33:02.352903 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.352908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.353106 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.353561 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:02.353706 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.354902 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.357127 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.358015 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.359330 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.359475 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.359528 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.359634 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":106,"threads_started":1}
I20260620 14:33:02.359920 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.350709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:41995 (local address 127.0.0.1:54170)
0620 14:33:02.351223 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.351236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:02.351321 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:02.352532 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:02.352540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:02.352570 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:02.352786 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.352797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.354329 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.354333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.357015 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.357029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.359435 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.359440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.359553 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.359581 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:33:02.359661 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:33:02.359739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":92,"thread_start_us":104,"threads_started":1}
I20260620 14:33:02.360534 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.350859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.4.58.67:41995)
0620 14:33:02.351532 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.351539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.351562 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:33:02.352075 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.352081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.352330 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.352486 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:02.352962 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.354178 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.357180 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.359085 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.359203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.359368 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.360246 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.360373 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":93,"threads_started":1}
I20260620 14:33:02.363628 17318 tablet_service.cc:1467] Processing CreateTablet for tablet 72029fe2d0654d6291c056c26ef06e36 (DEFAULT_TABLE table=test-table [id=2e0845b1f22948929e133b4701383408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:02.364005 17470 tablet_service.cc:1467] Processing CreateTablet for tablet 72029fe2d0654d6291c056c26ef06e36 (DEFAULT_TABLE table=test-table [id=2e0845b1f22948929e133b4701383408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:02.365247 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72029fe2d0654d6291c056c26ef06e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:02.366417 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72029fe2d0654d6291c056c26ef06e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:02.379621 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.351644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41901 (local address 127.0.0.1:48752)
0620 14:33:02.373554 (+ 21910us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.373575 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:02.373759 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:02.374548 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:02.374551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:02.374573 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:02.374692 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.374698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.376163 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.376166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.377464 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.377477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.379135 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.379139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.379244 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.379270 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:33:02.379354 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:33:02.379439 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3912,"thread_start_us":83,"threads_started":1}
I20260620 14:33:02.379990 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.351787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.4.58.65:41901)
0620 14:33:02.355167 (+ 3380us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.355173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.373875 (+ 18702us) server_negotiation.cc:407] Connection header received
0620 14:33:02.373968 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.373974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.374198 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.374347 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:02.374823 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.376020 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.377625 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.378819 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.378918 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.379066 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.379437 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.379618 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3269,"thread_start_us":64,"threads_started":1}
I20260620 14:33:02.382539 17570 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Bootstrap starting.
I20260620 14:33:02.383433 17200 tablet_service.cc:1467] Processing CreateTablet for tablet 72029fe2d0654d6291c056c26ef06e36 (DEFAULT_TABLE table=test-table [id=2e0845b1f22948929e133b4701383408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:02.384693 17200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72029fe2d0654d6291c056c26ef06e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:02.385200 17570 tablet_bootstrap.cc:654] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:02.386127 17570 log.cc:824] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:02.386979 17569 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Bootstrap starting.
I20260620 14:33:02.390367 17569 tablet_bootstrap.cc:654] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:02.391495 17569 log.cc:824] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:02.393960 17570 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: No bootstrap required, opened a new log
I20260620 14:33:02.394197 17570 ts_tablet_manager.cc:1397] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 14:33:02.397887 17573 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Bootstrap starting.
I20260620 14:33:02.400261 17569 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: No bootstrap required, opened a new log
I20260620 14:33:02.400499 17569 ts_tablet_manager.cc:1397] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260620 14:33:02.400666 17573 tablet_bootstrap.cc:654] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:02.401618 17573 log.cc:824] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:02.404983 17569 raft_consensus.cc:348] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.405332 17569 raft_consensus.cc:374] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:02.405400 17569 raft_consensus.cc:729] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89c63e947c2446f8398694f33bde748, State: Initialized, Role: FOLLOWER
I20260620 14:33:02.406001 17569 consensus_queue.cc:260] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.407286 17569 ts_tablet_manager.cc:1428] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260620 14:33:02.407725 17403 heartbeater.cc:503] Master 127.4.58.126:44067 was elected leader, sending a full tablet report...
I20260620 14:33:02.407620 17570 raft_consensus.cc:348] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.407927 17570 raft_consensus.cc:374] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:02.408053 17570 raft_consensus.cc:729] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a6e8deaaa0478882f461fdb211224a, State: Initialized, Role: FOLLOWER
I20260620 14:33:02.408594 17570 consensus_queue.cc:260] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.409739 17570 ts_tablet_manager.cc:1428] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260620 14:33:02.410127 17536 heartbeater.cc:503] Master 127.4.58.126:44067 was elected leader, sending a full tablet report...
I20260620 14:33:02.417631 17573 tablet_bootstrap.cc:492] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: No bootstrap required, opened a new log
I20260620 14:33:02.417913 17573 ts_tablet_manager.cc:1397] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260620 14:33:02.422478 17573 raft_consensus.cc:348] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.422822 17573 raft_consensus.cc:374] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:02.422880 17573 raft_consensus.cc:729] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849f5723f3a441d18b8226900e444166, State: Initialized, Role: FOLLOWER
I20260620 14:33:02.423456 17573 consensus_queue.cc:260] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.424206 17268 heartbeater.cc:503] Master 127.4.58.126:44067 was elected leader, sending a full tablet report...
I20260620 14:33:02.424659 17573 ts_tablet_manager.cc:1428] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 14:33:02.473465 17575 raft_consensus.cc:482] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:02.473739 17575 raft_consensus.cc:504] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.475134 17575 leader_election.cc:288] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849f5723f3a441d18b8226900e444166 (127.4.58.65:41901), b2a6e8deaaa0478882f461fdb211224a (127.4.58.67:41995)
I20260620 14:33:02.480813 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.476133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:50451 (local address 127.4.58.65:41901)
0620 14:33:02.476267 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.476271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.476289 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:33:02.476348 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.476351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.476405 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.476494 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:02.476871 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.477619 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.479274 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.479896 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.479986 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.480065 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.480526 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.480643 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:33:02.481258 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.475853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:56925 (local address 127.4.58.67:41995)
0620 14:33:02.475969 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.475973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.476880 (+ 907us) server_negotiation.cc:407] Connection header received
0620 14:33:02.477109 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.477112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.477165 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.477209 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:02.477650 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.478449 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.479908 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.480532 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.480614 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.480680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.481058 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.481140 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:33:02.481949 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.475384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41901 (local address 127.4.58.66:50451)
0620 14:33:02.476114 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.476129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:02.476242 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:02.476518 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:02.476521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:02.476541 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:02.476697 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.476704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.477798 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.477801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.479133 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.479144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.480201 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.480205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.480302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.480338 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:33:02.481695 (+ 1357us) client_negotiation.cc:240] Negotiation successful
0620 14:33:02.481782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260620 14:33:02.482822 17221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72029fe2d0654d6291c056c26ef06e36" candidate_uuid: "a89c63e947c2446f8398694f33bde748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849f5723f3a441d18b8226900e444166" is_pre_election: true
I20260620 14:33:02.483175 17221 raft_consensus.cc:2454] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89c63e947c2446f8398694f33bde748 in term 0.
I20260620 14:33:02.483676 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.475455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:41995 (local address 127.4.58.66:56925)
0620 14:33:02.476798 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.476812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:02.476984 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:02.477343 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:02.477346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:02.477363 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:02.477499 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.477506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.478588 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.478591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.479765 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:02.479776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.480807 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.480811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:02.480916 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.480947 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:33:02.482485 (+ 1538us) client_negotiation.cc:240] Negotiation successful
0620 14:33:02.483529 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":119,"threads_started":1}
I20260620 14:33:02.483739 17290 leader_election.cc:302] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849f5723f3a441d18b8226900e444166, a89c63e947c2446f8398694f33bde748; no voters:
I20260620 14:33:02.484138 17575 raft_consensus.cc:2781] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:02.484210 17575 raft_consensus.cc:482] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:02.484244 17575 raft_consensus.cc:3037] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:02.484611 17490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72029fe2d0654d6291c056c26ef06e36" candidate_uuid: "a89c63e947c2446f8398694f33bde748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a6e8deaaa0478882f461fdb211224a" is_pre_election: true
I20260620 14:33:02.484947 17490 raft_consensus.cc:2454] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89c63e947c2446f8398694f33bde748 in term 0.
I20260620 14:33:02.485525 17575 raft_consensus.cc:504] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.485903 17575 leader_election.cc:288] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [CANDIDATE]: Term 1 election: Requested vote from peers 849f5723f3a441d18b8226900e444166 (127.4.58.65:41901), b2a6e8deaaa0478882f461fdb211224a (127.4.58.67:41995)
I20260620 14:33:02.486317 17221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72029fe2d0654d6291c056c26ef06e36" candidate_uuid: "a89c63e947c2446f8398694f33bde748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849f5723f3a441d18b8226900e444166"
I20260620 14:33:02.486444 17221 raft_consensus.cc:3037] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:02.486492 17490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72029fe2d0654d6291c056c26ef06e36" candidate_uuid: "a89c63e947c2446f8398694f33bde748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a6e8deaaa0478882f461fdb211224a"
I20260620 14:33:02.486606 17490 raft_consensus.cc:3037] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:02.487689 17221 raft_consensus.cc:2454] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89c63e947c2446f8398694f33bde748 in term 1.
I20260620 14:33:02.487890 17490 raft_consensus.cc:2454] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89c63e947c2446f8398694f33bde748 in term 1.
I20260620 14:33:02.488221 17290 leader_election.cc:302] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849f5723f3a441d18b8226900e444166, a89c63e947c2446f8398694f33bde748; no voters:
I20260620 14:33:02.488399 17575 raft_consensus.cc:2781] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:02.488626 17575 raft_consensus.cc:686] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [term 1 LEADER]: Becoming Leader. State: Replica: a89c63e947c2446f8398694f33bde748, State: Running, Role: LEADER
I20260620 14:33:02.489012 17575 consensus_queue.cc:237] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } }
I20260620 14:33:02.492594 17070 catalog_manager.cc:5495] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 reported cstate change: term changed from 0 to 1, leader changed from <none> to a89c63e947c2446f8398694f33bde748 (127.4.58.66). New cstate: current_term: 1 leader_uuid: "a89c63e947c2446f8398694f33bde748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89c63e947c2446f8398694f33bde748" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 34957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 } health_report { overall_health: UNKNOWN } } }
W20260620 14:33:02.510123 17269 tablet.cc:2369] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:02.522863 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.517509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.4.58.66:34957)
0620 14:33:02.517648 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:02.517663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:02.517705 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:33:02.517895 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:02.517898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:02.517970 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:02.518077 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:02.518363 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.519159 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.520421 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:02.520794 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:02.520895 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:02.521370 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:02.522012 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:02.522137 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:02.522623 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 14:33:02.522691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:33:02.529222 17404 tablet.cc:2369] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:02.530980 17221 raft_consensus.cc:1261] T 72029fe2d0654d6291c056c26ef06e36 P 849f5723f3a441d18b8226900e444166 [term 1 FOLLOWER]: Refusing update from remote peer a89c63e947c2446f8398694f33bde748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:02.530982 17490 raft_consensus.cc:1261] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 1 FOLLOWER]: Refusing update from remote peer a89c63e947c2446f8398694f33bde748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:02.531733 17575 consensus_queue.cc:1035] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a6e8deaaa0478882f461fdb211224a" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 41995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:02.531929 17580 consensus_queue.cc:1035] T 72029fe2d0654d6291c056c26ef06e36 P a89c63e947c2446f8398694f33bde748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849f5723f3a441d18b8226900e444166" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:02.541949 17583 mvcc.cc:205] Tried to move back new op lower bound from 7298932664436264960 to 7298932664278310912. Current Snapshot: MvccSnapshot[applied={T|T < 7298932664436264960}]
W20260620 14:33:02.545121 17537 tablet.cc:2369] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:02.547803 17586 mvcc.cc:205] Tried to move back new op lower bound from 7298932664436264960 to 7298932664278310912. Current Snapshot: MvccSnapshot[applied={T|T < 7298932664436264960}]
I20260620 14:33:02.553536 17585 mvcc.cc:205] Tried to move back new op lower bound from 7298932664436264960 to 7298932664278310912. Current Snapshot: MvccSnapshot[applied={T|T < 7298932664436264960}]
I20260620 14:33:02.559140 17069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:02.559325 17069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:02.562119 17069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beef2bb883bd428e832f4c4404d1fa95: Sending DeleteTablet for 3 replicas of tablet 72029fe2d0654d6291c056c26ef06e36
I20260620 14:33:02.563105 17200 tablet_service.cc:1514] Processing DeleteTablet for tablet 72029fe2d0654d6291c056c26ef06e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:02 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48752
I20260620 14:33:02.563153 17318 tablet_service.cc:1514] Processing DeleteTablet for tablet 72029fe2d0654d6291c056c26ef06e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:02 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35702
I20260620 14:33:02.563603 17470 tablet_service.cc:1514] Processing DeleteTablet for tablet 72029fe2d0654d6291c056c26ef06e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:02 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:54170
I20260620 14:33:02.564374 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17108
W20260620 14:33:02.574452 17055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:41901 (error 108)
I20260620 14:33:02.576077 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.575716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41901 (local address 127.0.0.1:48768)
0620 14:33:02.575907 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.575983 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:41901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260620 14:33:02.576404 17055 catalog_manager.cc:4584] TS 849f5723f3a441d18b8226900e444166 (127.4.58.65:41901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72029fe2d0654d6291c056c26ef06e36: Network error: Client connection negotiation failed: client connection to 127.4.58.65:41901: connect: Connection refused (error 111)
I20260620 14:33:02.576815 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17272
I20260620 14:33:02.584524 17596 tablet_replica.cc:330] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a: stopping tablet replica
I20260620 14:33:02.585067 17596 raft_consensus.cc:2229] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:02.585616 17596 raft_consensus.cc:2258] T 72029fe2d0654d6291c056c26ef06e36 P b2a6e8deaaa0478882f461fdb211224a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:02.587495 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17406
I20260620 14:33:02.590804 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:02.590477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:34957 (local address 127.0.0.1:35720)
0620 14:33:02.590637 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:02.590704 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:34957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:33:02.595422 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17037
I20260620 14:33:02.603166 4329 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:33:02.938673 4329 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:33:02.938786 4329 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:33:03.274266 4329 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:33:03.274 UTC [16104] LOG: received smart shutdown request
2026-06-20 14:33:03.278 UTC [16104] LOG: background worker "logical replication launcher" (PID 16113) exited with exit code 1
2026-06-20 14:33:03.278 UTC [16108] LOG: shutting down
2026-06-20 14:33:03.446 UTC [16104] LOG: database system is shut down
2026-06-20T14:33:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46592 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:03 dist-test-slave-gqzk krb5kdc[17606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:03 dist-test-slave-gqzk krb5kdc[17606](info): set up 2 sockets
Jun 20 14:33:03 dist-test-slave-gqzk krb5kdc[17606](info): commencing operation
krb5kdc: starting...
W20260620 14:33:04.504592 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:33:04 dist-test-slave-gqzk krb5kdc[17606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:33:04.627839 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39037
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:04.739830 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:04.740170 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:04.740232 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:04.744436 17619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:04.744518 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:04.744546 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:04.744567 17619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:04.744586 17619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:04.750311 17619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:04.751515 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:04.752600 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:04.761812 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:04.761987 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:04.761862 17619 server_base.cc:1034] running on GCE node
W20260620 14:33:04.761835 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:04.762806 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:04.764031 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:04.765215 17619 hybrid_clock.cc:648] HybridClock initialized: now 1781965984765199 us; error 31 us; skew 500 ppm
Jun 20 14:33:04 dist-test-slave-gqzk krb5kdc[17606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:04.768211 17619 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:04.769223 17619 webserver.cc:460] Webserver started at http://127.4.58.126:45013/ using document root <none> and password file <none>
I20260620 14:33:04.769795 17619 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:04.769860 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:04.770071 17619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:04.772214 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "a350a0f962474cda853a29dfb6f8ec66"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "66ca5139997fa02755baccd5ee129ded"
server_key_iv: "b2b5498f4a34b7adf5dad52dfa5c0128"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.772848 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "a350a0f962474cda853a29dfb6f8ec66"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "66ca5139997fa02755baccd5ee129ded"
server_key_iv: "b2b5498f4a34b7adf5dad52dfa5c0128"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.776746 17619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:33:04.779644 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:04.781314 17619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:04.781471 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "a350a0f962474cda853a29dfb6f8ec66"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "66ca5139997fa02755baccd5ee129ded"
server_key_iv: "b2b5498f4a34b7adf5dad52dfa5c0128"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.781595 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:04.796947 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:04.800499 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:04.800716 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:04.808470 17619 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:39037
I20260620 14:33:04.808478 17686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:39037 every 8 connection(s)
I20260620 14:33:04.809542 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:04.812403 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:04.814450 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17619
I20260620 14:33:04.814575 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:33:04.814865 4329 external_mini_cluster.cc:1428] Setting key 4ce07b13b3558a0d7f90e6ffc438b7c7
I20260620 14:33:04.816475 17687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.58.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.125@KRBTEST.COM" created.
Jun 20 14:33:04 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:04.832207 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.818960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39037 (local address 127.0.0.1:49892)
0620 14:33:04.819598 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.819676 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:04.820506 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:04.821369 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:04.821374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:04.821724 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:04.822045 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:04.822059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:04.823327 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:04.823330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:04.824137 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:04.824147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:04.824888 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:04.824893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:04.824989 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:04.825623 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:04.825644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:04.827255 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:04.829270 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:04.829280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:04.829291 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:04.829601 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:04.829902 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:04.829906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:04.829908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:04.830064 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:04.830677 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:04.830686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:04.830865 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:33:04.831054 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 14:33:04.831464 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":74,"spinlock_wait_cycles":49024,"thread_start_us":153,"threads_started":1}
I20260620 14:33:04.832206 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.818960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.4.58.126:39037)
0620 14:33:04.819486 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:04.819506 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:04.819804 (+ 298us) server_negotiation.cc:407] Connection header received
0620 14:33:04.820875 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:04.820882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:04.821155 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:04.821375 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:04.822199 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:04.823202 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:04.824243 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:04.824699 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:04.824898 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:04.827418 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:04.827426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:04.827442 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:04.827469 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:04.829153 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:04.829701 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:04.829705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:04.829706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:04.829805 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:04.830164 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:04.830166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:04.830167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:04.830550 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:04.830643 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:04.831127 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:33:04.831363 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":350,"thread_start_us":147,"threads_started":1}
I20260620 14:33:04.834659 17687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:04.835423 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.835159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53488)
0620 14:33:04.835284 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.835358 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:33:04.835650 17637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
I20260620 14:33:04.836541 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.836345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53494)
0620 14:33:04.836444 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.836488 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:33:04.836730 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:45463: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
I20260620 14:33:04.842054 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:45463
--webserver_interface=127.4.58.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:04.886022 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } attempt: 1
I20260620 14:33:04.886840 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.886542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53498)
0620 14:33:04.886703 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.886764 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:33:04.887540 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.887338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53504)
0620 14:33:04.887440 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.887484 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:33:04.887678 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:45463: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
W20260620 14:33:04.954406 17694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:04.954700 17694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:04.954758 17694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:33:04.955830 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } attempt: 2
I20260620 14:33:04.956672 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.956374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53518)
0620 14:33:04.956525 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.956594 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:33:04.957386 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:04.957159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53530)
0620 14:33:04.957267 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:04.957316 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:33:04.957895 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:45463: Network error: Client connection negotiation failed: client connection to 127.4.58.125:45463: connect: Connection refused (error 111)
W20260620 14:33:04.959234 17694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:04.959291 17694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:04.959321 17694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:04.959352 17694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:04.959374 17694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:04.964988 17694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.58.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:04.966157 17694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:04.967183 17694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:04.975903 17694 server_base.cc:1034] running on GCE node
W20260620 14:33:04.975844 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:04.975793 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:04.975793 17700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:04.976717 17694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:04.977942 17694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:04.979110 17694 hybrid_clock.cc:648] HybridClock initialized: now 1781965984979087 us; error 42 us; skew 500 ppm
Jun 20 14:33:04 dist-test-slave-gqzk krb5kdc[17606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:04.982012 17694 init.cc:377] Logged in from keytab as kudu/127.4.58.125@KRBTEST.COM (short username kudu)
I20260620 14:33:04.982913 17694 webserver.cc:460] Webserver started at http://127.4.58.125:32973/ using document root <none> and password file <none>
I20260620 14:33:04.983457 17694 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:04.983526 17694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:04.983697 17694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:04.985915 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data/instance:
uuid: "089dab8103c843c28c0ef18414fe2308"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "21ce7fe27eeafec0ffd02941bd04c3b9"
server_key_iv: "98857880b7c0990b7f2b12193b7ce2fc"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.986560 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal/instance:
uuid: "089dab8103c843c28c0ef18414fe2308"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "21ce7fe27eeafec0ffd02941bd04c3b9"
server_key_iv: "98857880b7c0990b7f2b12193b7ce2fc"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.990440 17694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:33:04.993170 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:04.994495 17694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:33:04.994614 17694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "089dab8103c843c28c0ef18414fe2308"
format_stamp: "Formatted at 2026-06-20 14:33:04 on dist-test-slave-gqzk"
server_key: "21ce7fe27eeafec0ffd02941bd04c3b9"
server_key_iv: "98857880b7c0990b7f2b12193b7ce2fc"
server_key_version: "encryptionkey@0"
I20260620 14:33:04.994717 17694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:05.016173 17694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:05.019383 17694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:05.019548 17694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:05.026559 17694 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.125:45463
I20260620 14:33:05.026582 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.125:45463 every 8 connection(s)
I20260620 14:33:05.027532 17694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
I20260620 14:33:05.028578 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17694
I20260620 14:33:05.028715 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-1/wal/instance
I20260620 14:33:05.029045 4329 external_mini_cluster.cc:1428] Setting key 0be455c854c0d4ead5fa036b972ee993
I20260620 14:33:05.030673 17762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:05.034847 17762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.58.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.124@KRBTEST.COM" created.
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:05.049062 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } attempt: 3
I20260620 14:33:05.050127 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.036918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.4.58.126:39037)
0620 14:33:05.037134 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.037138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.038010 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:33:05.039047 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.039050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.039106 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.039186 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.040537 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.041175 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.042340 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.042584 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.042678 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.045532 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.045536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.045538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.045567 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.047577 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.048180 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.048184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.048186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.048268 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.048708 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.048711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.048712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.048950 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.049033 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.049812 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.049954 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:33:05.051539 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.037291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39037 (local address 127.0.0.1:49906)
0620 14:33:05.037820 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.037887 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.038840 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.039374 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.039381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.039719 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.040368 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.040388 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.041318 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.041320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.042211 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.042220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.042743 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.042747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.042908 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.043483 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.043500 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.045292 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.047715 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.047723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.047775 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.048065 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.048410 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.048413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.048415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.048592 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.049037 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.049043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.049481 (+ 438us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.050284 (+ 803us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.050650 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":125,"threads_started":1}
I20260620 14:33:05.053211 17762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
I20260620 14:33:05.056205 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:42453
--webserver_interface=127.4.58.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
I20260620 14:33:05.060307 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.049653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53538 (local address 127.4.58.125:45463)
0620 14:33:05.050056 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.050067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.050090 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:33:05.050132 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.050139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.050403 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.050541 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.050924 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.051816 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.052733 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.053257 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.053357 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.055677 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.055688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.055704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.055729 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.058075 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.058520 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.058527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.058529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.058637 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.059272 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.059274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.059275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.059520 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.059637 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.059965 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.060107 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":182,"threads_started":1}
I20260620 14:33:05.060452 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.049540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53538)
0620 14:33:05.049727 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.049768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.049852 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.050533 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.050535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.050677 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.050813 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.050821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.051939 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.051942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.052631 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.052641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.053376 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.053380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.053486 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.054088 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.054102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.055530 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.058190 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.058194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.058197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.058399 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.058744 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.058748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.058750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.058863 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.059631 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.059634 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.059750 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.059863 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.060012 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:33:05.062516 17687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:05.063232 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.062976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34384)
0620 14:33:05.063095 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.063149 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:33:05.063489 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.053709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53540)
0620 14:33:05.053821 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.053835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.053904 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.054421 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.054423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.054579 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.054722 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.054728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.055773 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.055776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.056545 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.056556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.057118 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.057122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.057203 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.057651 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.057678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.059181 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.061927 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.061930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.061941 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.062212 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.062508 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.062510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.062512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.062587 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.062997 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.062999 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.063080 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.063195 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.063327 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:33:05.063539 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.053867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.4.58.125:45463)
0620 14:33:05.054207 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.054216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.054228 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:33:05.054262 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.054265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.054305 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.054393 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.054882 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.055667 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.056662 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.056996 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.057096 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.059308 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.059311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.059314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.059338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.061823 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.062317 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.062321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.062323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.062408 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.062674 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.062677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.062679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.062878 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.062965 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.063273 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.063388 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260620 14:33:05.064073 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.063799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34390)
0620 14:33:05.063976 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.064020 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260620 14:33:05.064190 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
I20260620 14:33:05.064584 17762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:05.065256 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.064970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34392)
0620 14:33:05.065090 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.065187 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:33:05.065505 17711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
I20260620 14:33:05.066396 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.066179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34398)
0620 14:33:05.066286 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.066337 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:33:05.066570 17762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
I20260620 14:33:05.095355 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 1
I20260620 14:33:05.096227 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.095877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34410)
0620 14:33:05.096027 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.096093 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:33:05.096998 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.096798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34420)
0620 14:33:05.096893 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.096940 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:33:05.097128 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
I20260620 14:33:05.115847 17762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 1
I20260620 14:33:05.116780 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.116408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34422)
0620 14:33:05.116580 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.116658 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:33:05.117504 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.117288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34436)
0620 14:33:05.117389 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.117443 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:33:05.117690 17762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
W20260620 14:33:05.171722 17769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:05.172134 17769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:05.172204 17769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:33:05.172266 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 2
I20260620 14:33:05.173233 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.172897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34446)
0620 14:33:05.173081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.173150 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:33:05.174103 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.173820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34458)
0620 14:33:05.173982 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.174039 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 14:33:05.174232 17687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
W20260620 14:33:05.176877 17769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:05.176944 17769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:05.176977 17769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:05.177013 17769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:05.177047 17769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:05.182585 17769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39037,127.4.58.125:45463,127.4.58.124:42453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.58.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:05.183761 17769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:05.184917 17769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:05.185889 17762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 2
I20260620 14:33:05.186769 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.186443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34464)
0620 14:33:05.186622 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.186689 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 14:33:05.187543 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.187296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34466)
0620 14:33:05.187413 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.187463 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:33:05.187695 17762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:42453: Network error: Client connection negotiation failed: client connection to 127.4.58.124:42453: connect: Connection refused (error 111)
W20260620 14:33:05.194218 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:05.194228 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:05.194235 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:05.194391 17769 server_base.cc:1034] running on GCE node
I20260620 14:33:05.195056 17769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:05.196190 17769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:05.197356 17769 hybrid_clock.cc:648] HybridClock initialized: now 1781965985197335 us; error 36 us; skew 500 ppm
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:05.200105 17769 init.cc:377] Logged in from keytab as kudu/127.4.58.124@KRBTEST.COM (short username kudu)
I20260620 14:33:05.201007 17769 webserver.cc:460] Webserver started at http://127.4.58.124:44653/ using document root <none> and password file <none>
I20260620 14:33:05.201535 17769 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:05.201622 17769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:05.201810 17769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:05.203882 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data/instance:
uuid: "007b3e90f2824cb8ac9c9f7fa508d3be"
format_stamp: "Formatted at 2026-06-20 14:33:05 on dist-test-slave-gqzk"
server_key: "25913309153550a062499866ed378887"
server_key_iv: "9644bd56602bd41361586185c00e23d6"
server_key_version: "encryptionkey@0"
I20260620 14:33:05.204413 17769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal/instance:
uuid: "007b3e90f2824cb8ac9c9f7fa508d3be"
format_stamp: "Formatted at 2026-06-20 14:33:05 on dist-test-slave-gqzk"
server_key: "25913309153550a062499866ed378887"
server_key_iv: "9644bd56602bd41361586185c00e23d6"
server_key_version: "encryptionkey@0"
I20260620 14:33:05.207971 17769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:33:05.210426 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:05.211653 17769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:33:05.211776 17769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "007b3e90f2824cb8ac9c9f7fa508d3be"
format_stamp: "Formatted at 2026-06-20 14:33:05 on dist-test-slave-gqzk"
server_key: "25913309153550a062499866ed378887"
server_key_iv: "9644bd56602bd41361586185c00e23d6"
server_key_version: "encryptionkey@0"
I20260620 14:33:05.211872 17769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:05.238886 17769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:05.241897 17769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:05.242074 17769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:05.248631 17769 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.124:42453
I20260620 14:33:05.248687 17837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.124:42453 every 8 connection(s)
I20260620 14:33:05.249588 17769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
I20260620 14:33:05.252316 17838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:05.253808 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17769
I20260620 14:33:05.253939 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965824702348-4329-0/minicluster-data/master-2/wal/instance
I20260620 14:33:05.254261 4329 external_mini_cluster.cc:1428] Setting key 0fbb19233f1f7a8a4863b24cc71da2ad
I20260620 14:33:05.257320 17838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:05.268782 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.259414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.4.58.126:39037)
0620 14:33:05.259546 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.259550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.259808 (+ 258us) server_negotiation.cc:407] Connection header received
0620 14:33:05.259930 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.259934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.259992 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.260080 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.261024 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.261770 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.263051 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.263280 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.263360 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.265731 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.265735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.265737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.265765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.267017 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267462 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.267465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.267467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.267544 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267793 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.267795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.267796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.267985 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.268072 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.268489 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.268610 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:33:05.269794 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.259300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39037 (local address 127.0.0.1:49914)
0620 14:33:05.259677 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.259726 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.259845 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.260157 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.260162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.260630 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.260808 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.260816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.261921 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.261923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.262923 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.262934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.263438 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.263443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.263546 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.264125 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.264142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.265589 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.267124 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.267140 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.267378 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.267627 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.267631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.267711 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.268095 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.268102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.268245 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.268407 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.268776 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":142,"thread_start_us":94,"threads_started":1}
I20260620 14:33:05.269794 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.255691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.4.58.124:42453)
0620 14:33:05.256133 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.256152 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.256198 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:33:05.257050 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.257076 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.257531 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.257985 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.258700 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.259983 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.261141 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.261990 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.262133 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.264853 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.264859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.264871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.264893 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.266588 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267011 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.267017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.267022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.267140 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.267416 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.267420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.267422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.267953 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.268134 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.268367 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.268692 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260620 14:33:05.271819 17838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:05.273388 17687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 3
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
I20260620 14:33:05.278875 17762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } attempt: 3
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:33:05.282231 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.272283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:45463 (local address 127.0.0.1:53552)
0620 14:33:05.272448 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.272461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.272536 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.272805 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.272807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.272977 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.273074 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.273079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.273901 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.273904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.275124 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.275131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.275825 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.275829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.275925 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.277004 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.277017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.278364 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.280167 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.280170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.280172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.280711 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.281045 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.281048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.281050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.281351 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.281780 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.281783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.281868 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.281947 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.282061 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2}
I20260620 14:33:05.282374 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.272387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.4.58.126:39037)
0620 14:33:05.272485 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.272489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.272501 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:33:05.272626 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.272629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.272674 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.272736 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.273334 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.273870 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.274775 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.274994 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.275080 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.277387 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.277391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.277394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.277419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.280085 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.280654 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.280658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.280660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.280737 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.281034 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.281038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.281040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.281273 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.281498 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.282091 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.282226 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:33:05.282812 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.272381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.4.58.125:45463)
0620 14:33:05.272490 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.272495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.272508 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:33:05.272637 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.272640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.272687 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.272786 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.273184 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.273798 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.275238 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.275604 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.275678 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.278498 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.278502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.278505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.278534 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.280031 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.280861 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.280864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.280866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.280951 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.281441 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.281444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.281446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.281656 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.281737 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.282543 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.282669 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:33:05.283465 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.274223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34474)
0620 14:33:05.274320 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.274334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.274404 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.274696 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.274699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.274875 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.274967 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.274972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.276313 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.276317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.276858 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.276867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.277416 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.277420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.277501 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.278010 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.278020 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.279398 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.282080 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.282084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.282086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.282322 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.282590 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.282594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.282596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.282693 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.283077 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.283080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.283169 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.283239 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.283333 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:33:05.283982 17838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:33:05.284799 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.274146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.4.58.124:42453)
0620 14:33:05.274511 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.274514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.274523 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:33:05.274546 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.274548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.274589 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.274647 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.275096 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.276190 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.276961 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.277288 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.277421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.280203 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.280206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.280209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.280230 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.281871 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.282427 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.282430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.282432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.282499 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.282793 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.282795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.282797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.282976 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.283707 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.283780 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.283882 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":306,"spinlock_wait_cycles":768}
I20260620 14:33:05.288304 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.279477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34484)
0620 14:33:05.279652 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.279663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.279723 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.280114 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.280116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.280259 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.280337 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.280342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.281362 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.281365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.282107 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.282116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.282807 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.282810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.282899 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.283396 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.283409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.284845 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.286711 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.286714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.286716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.286946 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.287316 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.287319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.287321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.287408 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.287830 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.287832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.287923 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.288026 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.288157 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 14:33:05.288388 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.279365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.4.58.124:42453)
0620 14:33:05.279919 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.279925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.279935 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:33:05.279958 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.279960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.279990 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.280059 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.280464 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.281244 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.282232 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.282679 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.283981 (+ 1302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.285003 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.285007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.285009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.285031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.286593 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.287045 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.287048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.287123 (+ 75us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.287210 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.287502 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.287505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.287507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.287720 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.287816 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.288098 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.288211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":117,"threads_started":1}
I20260620 14:33:05.288601 17687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: Bootstrap starting.
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
I20260620 14:33:05.291188 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: Bootstrap starting.
I20260620 14:33:05.291635 17687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:05.292603 17687 log.cc:824] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:05.293569 17762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:05.294332 17762 log.cc:824] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:05.294750 17687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: No bootstrap required, opened a new log
Jun 20 14:33:05 dist-test-slave-gqzk krb5kdc[17606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965985, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:33:05.296368 17762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: No bootstrap required, opened a new log
I20260620 14:33:05.299190 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.285106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:42453 (local address 127.0.0.1:34492)
0620 14:33:05.289002 (+ 3896us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:05.289015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:05.289100 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:05.289320 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:05.289323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:05.289469 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:05.289555 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.289561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.290477 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.290480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.291179 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:05.291190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.291687 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.291690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:05.291763 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.292243 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:05.292256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:05.295526 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:05.297816 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.297820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.297822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.298041 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.298303 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:05.298307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:05.298308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:05.298374 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:05.298742 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:05.298745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:05.298819 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:33:05.298890 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:33:05.299019 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570}
I20260620 14:33:05.299362 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.284101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.4.58.125:45463)
0620 14:33:05.284220 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.284222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.284233 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:33:05.284314 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.284316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.284346 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.284402 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.285179 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.285850 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.286663 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.286968 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.287064 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.295736 (+ 8672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.295740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.295743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.295768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.297335 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.297869 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.297873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.297875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.297970 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.298335 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.298338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.298340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.298557 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.298692 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.299048 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.299194 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:33:05.299094 17687 raft_consensus.cc:348] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.299480 17687 raft_consensus.cc:374] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:05.299537 17687 raft_consensus.cc:729] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a350a0f962474cda853a29dfb6f8ec66, State: Initialized, Role: FOLLOWER
I20260620 14:33:05.299675 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.285520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.4.58.124:42453)
0620 14:33:05.285816 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.285821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.289179 (+ 3358us) server_negotiation.cc:407] Connection header received
0620 14:33:05.289202 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.289204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.289236 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.289642 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.289700 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.290375 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.291300 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.291576 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.292848 (+ 1272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.295695 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.295699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.295702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.295726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.297709 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.298151 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.298154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.298156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.298225 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.298456 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.298458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.298460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.298651 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.299372 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.299440 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.299540 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260620 14:33:05.300088 17687 consensus_queue.cc:260] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.301513 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.301687 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.302019 17687 sys_catalog.cc:564] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:05.302014 17762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.302268 17762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260620 14:33:05.302450 17640 tablet.cc:2369] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:05.302942 17838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: Bootstrap starting.
I20260620 14:33:05.302994 17762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 089dab8103c843c28c0ef18414fe2308, State: Initialized, Role: FOLLOWER
I20260620 14:33:05.303517 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:33:05.303679 17762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.305208 17762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:05.305776 17838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:05.306731 17838 log.cc:824] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: Log is configured to *not* fsync() on all Append() calls
W20260620 14:33:05.308223 17859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:05.308305 17859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:05.310606 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.310755 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.315553 17838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: No bootstrap required, opened a new log
I20260620 14:33:05.319025 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.307636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.4.58.124:42453)
0620 14:33:05.307757 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.307761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.309116 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 14:33:05.309224 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.309227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.309277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.309354 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.310296 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.311032 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.312288 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.312615 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.312697 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.314790 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.314794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.314797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.314826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.316687 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.317313 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.317317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.317319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.317409 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.317872 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.317876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.317878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.318211 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.318338 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.318711 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.318847 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:33:05.319785 17838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.320070 17838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:05.320135 17838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007b3e90f2824cb8ac9c9f7fa508d3be, State: Initialized, Role: FOLLOWER
I20260620 14:33:05.320677 17838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.312510 17619 master_runner.cc:186] Error getting master registration for 127.4.58.124:42453: OK, instance_id { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" instance_seqno: 1781965985246906 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:33:05.321771 17619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:33:05.322194 17838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:05.317731 17694 master_runner.cc:186] Error getting master registration for 127.4.58.124:42453: OK, instance_id { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" instance_seqno: 1781965985246906 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:33:05.322505 17694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:33:05.325860 17876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:33:05.329084 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.308129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.4.58.126:39037)
0620 14:33:05.308239 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.308244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.309400 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 14:33:05.309561 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.309564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.309605 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.309666 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.310880 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.311554 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.312593 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.312953 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.313022 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.325352 (+ 12330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.325357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.325360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.325392 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.326951 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.327406 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.327409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.327410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.327492 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.327797 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.327800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.327802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.328019 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.328143 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.328812 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.328936 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:33:05.330334 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:05.308699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.4.58.125:45463)
0620 14:33:05.308965 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:05.308969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:05.319981 (+ 11012us) server_negotiation.cc:407] Connection header received
0620 14:33:05.320360 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:05.320364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:05.320420 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:05.320471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:05.321495 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.322299 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.323623 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:05.323954 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:05.324042 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:05.326805 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:05.326809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:05.326812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:05.326839 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:05.328376 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.328852 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.328855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.328857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.328935 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:05.329438 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:05.329442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:05.329443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:05.329646 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:05.329743 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:05.330074 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:33:05.330187 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260620 14:33:05.330542 17769 master_runner.cc:263] No leader master found from master 007b3e90f2824cb8ac9c9f7fa508d3be
I20260620 14:33:05.330607 17769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:33:05.325942 17876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:05.333041 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.333220 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:33:05.342665 17892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:05.342768 17892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:05.417955 17847 raft_consensus.cc:482] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:05.418175 17847 raft_consensus.cc:504] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.419286 17847 leader_election.cc:288] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 089dab8103c843c28c0ef18414fe2308 (127.4.58.125:45463), 007b3e90f2824cb8ac9c9f7fa508d3be (127.4.58.124:42453)
I20260620 14:33:05.420004 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a350a0f962474cda853a29dfb6f8ec66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" is_pre_election: true
I20260620 14:33:05.420352 17812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a350a0f962474cda853a29dfb6f8ec66 in term 0.
I20260620 14:33:05.421121 17636 leader_election.cc:302] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 007b3e90f2824cb8ac9c9f7fa508d3be, a350a0f962474cda853a29dfb6f8ec66; no voters:
I20260620 14:33:05.421065 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a350a0f962474cda853a29dfb6f8ec66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089dab8103c843c28c0ef18414fe2308" is_pre_election: true
I20260620 14:33:05.421378 17736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a350a0f962474cda853a29dfb6f8ec66 in term 0.
I20260620 14:33:05.421432 17847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:05.421509 17847 raft_consensus.cc:482] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:05.421557 17847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:05.422868 17847 raft_consensus.cc:504] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.423249 17847 leader_election.cc:288] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [CANDIDATE]: Term 1 election: Requested vote from peers 089dab8103c843c28c0ef18414fe2308 (127.4.58.125:45463), 007b3e90f2824cb8ac9c9f7fa508d3be (127.4.58.124:42453)
I20260620 14:33:05.423537 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a350a0f962474cda853a29dfb6f8ec66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "089dab8103c843c28c0ef18414fe2308"
I20260620 14:33:05.423648 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a350a0f962474cda853a29dfb6f8ec66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be"
I20260620 14:33:05.423697 17736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:05.423769 17812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:05.424986 17736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a350a0f962474cda853a29dfb6f8ec66 in term 1.
I20260620 14:33:05.424986 17812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a350a0f962474cda853a29dfb6f8ec66 in term 1.
I20260620 14:33:05.425320 17637 leader_election.cc:302] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 007b3e90f2824cb8ac9c9f7fa508d3be, 089dab8103c843c28c0ef18414fe2308, a350a0f962474cda853a29dfb6f8ec66; no voters:
I20260620 14:33:05.425559 17847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:05.425822 17847 raft_consensus.cc:686] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [term 1 LEADER]: Becoming Leader. State: Replica: a350a0f962474cda853a29dfb6f8ec66, State: Running, Role: LEADER
I20260620 14:33:05.426201 17847 consensus_queue.cc:237] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } }
I20260620 14:33:05.428102 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a350a0f962474cda853a29dfb6f8ec66. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.428236 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:05.441215 17896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:05.448058 17896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:05.456035 17812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [term 1 FOLLOWER]: Refusing update from remote peer a350a0f962474cda853a29dfb6f8ec66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:05.457036 17736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [term 1 FOLLOWER]: Refusing update from remote peer a350a0f962474cda853a29dfb6f8ec66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:05.457891 17894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:05.458725 17847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:05.466004 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: SysCatalogTable state changed. Reason: New leader a350a0f962474cda853a29dfb6f8ec66. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.466189 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.471503 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a350a0f962474cda853a29dfb6f8ec66. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.471697 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.477727 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.477913 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:05.479327 17853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.479478 17853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.479943 17894 sys_catalog.cc:455] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.480075 17894 sys_catalog.cc:458] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:05.481595 17880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a350a0f962474cda853a29dfb6f8ec66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a350a0f962474cda853a29dfb6f8ec66" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39037 } } peers { permanent_uuid: "089dab8103c843c28c0ef18414fe2308" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 45463 } } peers { permanent_uuid: "007b3e90f2824cb8ac9c9f7fa508d3be" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 42453 } } }
I20260620 14:33:05.481734 17880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:33:05.482349 17897 mvcc.cc:205] Tried to move back new op lower bound from 7298932676416512000 to 7298932676309032960. Current Snapshot: MvccSnapshot[applied={T|T < 7298932676416512000}]
I20260620 14:33:05.482542 17900 mvcc.cc:205] Tried to move back new op lower bound from 7298932676416512000 to 7298932676309032960. Current Snapshot: MvccSnapshot[applied={T|T < 7298932676416512000}]
I20260620 14:33:05.488705 17896 catalog_manager.cc:1347] Generated new cluster ID: bc7d5c4d4ce047949995cac2ef91e967
I20260620 14:33:05.488822 17896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:05.517961 17896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:05.519208 17896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:05.531561 17896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a350a0f962474cda853a29dfb6f8ec66: Generated new TSK 0
I20260620 14:33:05.532527 17896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:33:06.334110 17876 catalog_manager.cc:1259] Loaded cluster ID: bc7d5c4d4ce047949995cac2ef91e967
I20260620 14:33:06.334201 17876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: loading cluster ID for follower catalog manager: success
I20260620 14:33:06.335914 17876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: acquiring CA information for follower catalog manager: success
I20260620 14:33:06.337126 17876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 089dab8103c843c28c0ef18414fe2308: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:33:06.343979 17892 catalog_manager.cc:1259] Loaded cluster ID: bc7d5c4d4ce047949995cac2ef91e967
I20260620 14:33:06.344056 17892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: loading cluster ID for follower catalog manager: success
I20260620 14:33:06.345755 17892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: acquiring CA information for follower catalog manager: success
I20260620 14:33:06.346922 17892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 007b3e90f2824cb8ac9c9f7fa508d3be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:33:06.592702 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17619
I20260620 14:33:06.598196 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17694
I20260620 14:33:06.603523 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17769
2026-06-20T14:33:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3056 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:06 dist-test-slave-gqzk krb5kdc[17911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:06 dist-test-slave-gqzk krb5kdc[17911](info): set up 2 sockets
Jun 20 14:33:06 dist-test-slave-gqzk krb5kdc[17911](info): commencing operation
krb5kdc: starting...
W20260620 14:33:07.556813 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:33:07 dist-test-slave-gqzk krb5kdc[17911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:33:07.684060 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44867
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:44867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:07.795619 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:07.795922 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:07.795979 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:07.800133 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:07.800207 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:07.800235 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:07.800256 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:07.800274 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:07.805743 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:44867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:44867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:07.806850 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:07.807907 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:07.816946 17924 server_base.cc:1034] running on GCE node
W20260620 14:33:07.816903 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:07.816900 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:07.816900 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:07.817942 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:07.819193 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:07.820389 17924 hybrid_clock.cc:648] HybridClock initialized: now 1781965987820366 us; error 67 us; skew 500 ppm
Jun 20 14:33:07 dist-test-slave-gqzk krb5kdc[17911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965987, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:07.823630 17924 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:07.824673 17924 webserver.cc:460] Webserver started at http://127.4.58.126:40773/ using document root <none> and password file <none>
I20260620 14:33:07.825297 17924 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:07.825372 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:07.825584 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:07.827749 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "831a6da5521b46f08072dc33d2f5e3c7"
format_stamp: "Formatted at 2026-06-20 14:33:07 on dist-test-slave-gqzk"
server_key: "1ca761eac9aa5cbc0796ed417723ee61"
server_key_iv: "7826611d801dcf0e7bc2b085efff221d"
server_key_version: "encryptionkey@0"
I20260620 14:33:07.828338 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "831a6da5521b46f08072dc33d2f5e3c7"
format_stamp: "Formatted at 2026-06-20 14:33:07 on dist-test-slave-gqzk"
server_key: "1ca761eac9aa5cbc0796ed417723ee61"
server_key_iv: "7826611d801dcf0e7bc2b085efff221d"
server_key_version: "encryptionkey@0"
I20260620 14:33:07.832554 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 14:33:07.835304 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:07.836746 17924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:07.836946 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "831a6da5521b46f08072dc33d2f5e3c7"
format_stamp: "Formatted at 2026-06-20 14:33:07 on dist-test-slave-gqzk"
server_key: "1ca761eac9aa5cbc0796ed417723ee61"
server_key_iv: "7826611d801dcf0e7bc2b085efff221d"
server_key_version: "encryptionkey@0"
I20260620 14:33:07.837081 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:07.855553 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:07.859002 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:07.859202 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:07.866526 17924 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:44867
I20260620 14:33:07.866578 17991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:44867 every 8 connection(s)
I20260620 14:33:07.867584 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:07.870764 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 17924
I20260620 14:33:07.870890 17992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:07.870905 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:33:07.871201 4329 external_mini_cluster.cc:1428] Setting key 368d4bc0e38076962dbcc76b5d09c44b
I20260620 14:33:07.878474 17992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7: Bootstrap starting.
Jun 20 14:33:07 dist-test-slave-gqzk krb5kdc[17911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:07.881421 17992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:07.882306 17992 log.cc:824] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:07.884359 17992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7: No bootstrap required, opened a new log
I20260620 14:33:07.886404 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:07.872650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.4.58.126:44867)
0620 14:33:07.873147 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:07.873164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:07.873201 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:33:07.873884 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:07.873903 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:07.874268 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:07.874557 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:07.875490 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:07.876541 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:07.877807 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:07.878339 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:07.878481 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:07.881651 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:07.881657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:07.881673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:07.881710 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:07.883512 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:07.883968 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:07.883973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:07.883978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:07.884104 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:07.884383 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:07.884386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:07.884388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:07.884863 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:07.885054 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:07.885361 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:33:07.885659 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":134,"threads_started":1}
I20260620 14:33:07.888336 17992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } }
I20260620 14:33:07.888619 17992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:07.888684 17992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831a6da5521b46f08072dc33d2f5e3c7, State: Initialized, Role: FOLLOWER
I20260620 14:33:07.889213 17992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } }
I20260620 14:33:07.889385 17992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:07.889458 17992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:07.889545 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:33:07.889578 17992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:07.890604 17992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } }
I20260620 14:33:07.890997 17992 leader_election.cc:302] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831a6da5521b46f08072dc33d2f5e3c7; no voters:
I20260620 14:33:07.891378 17992 leader_election.cc:288] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:07.891607 17997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:07.891822 17997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [term 1 LEADER]: Becoming Leader. State: Replica: 831a6da5521b46f08072dc33d2f5e3c7, State: Running, Role: LEADER
I20260620 14:33:07.892218 17997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } }
I20260620 14:33:07.892722 17992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:07.893857 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "831a6da5521b46f08072dc33d2f5e3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } } }
I20260620 14:33:07.894006 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:07.894304 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831a6da5521b46f08072dc33d2f5e3c7. Latest consensus state: current_term: 1 leader_uuid: "831a6da5521b46f08072dc33d2f5e3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831a6da5521b46f08072dc33d2f5e3c7" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 44867 } } }
I20260620 14:33:07.894393 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:07.894400 18006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:07.897883 18006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:07.904309 18006 catalog_manager.cc:1347] Generated new cluster ID: b2a1db7072cb4d57bc6ffa95ec5dfee2
I20260620 14:33:07.904405 18006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:07.932874 18006 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:07.934283 18006 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:07.948395 18006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831a6da5521b46f08072dc33d2f5e3c7: Generated new TSK 0
I20260620 14:33:07.949381 18006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:33:07.997242 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 17924
2026-06-20T14:33:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1393 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:08 dist-test-slave-gqzk krb5kdc[18019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:08 dist-test-slave-gqzk krb5kdc[18019](info): set up 2 sockets
Jun 20 14:33:08 dist-test-slave-gqzk krb5kdc[18019](info): commencing operation
krb5kdc: starting...
W20260620 14:33:08.952900 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:33:09.077060 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:09.188271 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:09.188587 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:09.188645 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:09.192795 18032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:09.192878 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:09.192904 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:09.192927 18032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:09.192957 18032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:09.198457 18032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:36951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:09.199561 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:09.200639 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:09.209216 18040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.209229 18037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.209545 18038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:09.209398 18032 server_base.cc:1034] running on GCE node
I20260620 14:33:09.210273 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:09.211419 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:09.212607 18032 hybrid_clock.cc:648] HybridClock initialized: now 1781965989212590 us; error 40 us; skew 500 ppm
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:09.220273 18032 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:09.221349 18032 webserver.cc:460] Webserver started at http://127.4.58.126:46561/ using document root <none> and password file <none>
I20260620 14:33:09.221921 18032 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:09.221993 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:09.222205 18032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:09.224277 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "e754f05539bc43148b4926445964406f"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "bd3818b9160dcecac16f1d2c4a51a42c"
server_key_iv: "e08b87b53c78d9d065a12e200c146afa"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.224862 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "e754f05539bc43148b4926445964406f"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "bd3818b9160dcecac16f1d2c4a51a42c"
server_key_iv: "e08b87b53c78d9d065a12e200c146afa"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.228654 18032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:33:09.231233 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.232524 18032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:09.232657 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e754f05539bc43148b4926445964406f"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "bd3818b9160dcecac16f1d2c4a51a42c"
server_key_iv: "e08b87b53c78d9d065a12e200c146afa"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.232803 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:09.246415 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:09.249630 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:09.249815 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:09.256506 18032 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:36951
I20260620 14:33:09.256934 18099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:36951 every 8 connection(s)
I20260620 14:33:09.257550 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:09.260358 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:09.263924 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18032
I20260620 14:33:09.264031 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:33:09.264322 4329 external_mini_cluster.cc:1428] Setting key 971232933c27e4e0eb453706607b8e06
I20260620 14:33:09.266593 18100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap starting.
I20260620 14:33:09.269321 18100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:09.270171 18100 log.cc:824] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:09.272456 18100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: No bootstrap required, opened a new log
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:09.276520 18100 raft_consensus.cc:348] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:09.276799 18100 raft_consensus.cc:374] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:09.276870 18100 raft_consensus.cc:729] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e754f05539bc43148b4926445964406f, State: Initialized, Role: FOLLOWER
I20260620 14:33:09.277405 18100 consensus_queue.cc:260] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:09.277575 18100 raft_consensus.cc:388] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:09.277650 18100 raft_consensus.cc:482] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:09.277781 18100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:09.278841 18100 raft_consensus.cc:504] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:09.279181 18100 leader_election.cc:302] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e754f05539bc43148b4926445964406f; no voters:
I20260620 14:33:09.279218 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.265813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.4.58.126:36951)
0620 14:33:09.266257 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:09.266274 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:09.266314 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:33:09.266996 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:09.267017 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:09.267384 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:09.267683 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:09.268663 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.269789 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.270926 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.271434 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.271598 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:09.274331 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:09.274336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:09.274352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:09.274382 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:09.276061 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.276546 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.276551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.276556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.276685 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.276994 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.276998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.277000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.277560 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:09.277804 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:09.278076 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:33:09.278399 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
I20260620 14:33:09.279503 18100 leader_election.cc:288] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:09.279608 18105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:09.279817 18105 raft_consensus.cc:686] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 LEADER]: Becoming Leader. State: Replica: e754f05539bc43148b4926445964406f, State: Running, Role: LEADER
I20260620 14:33:09.280217 18105 consensus_queue.cc:237] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:09.280930 18100 sys_catalog.cc:564] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:09.282084 18107 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e754f05539bc43148b4926445964406f. Latest consensus state: current_term: 1 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:09.282218 18107 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:09.282392 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:09.282591 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
W20260620 14:33:09.285882 18120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:09.285974 18120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:09.286077 18116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:09.286783 18116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:09.292057 18116 catalog_manager.cc:1347] Generated new cluster ID: b1d9343b43ad4f8f809b5da1e976a06f
I20260620 14:33:09.292135 18116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:09.314352 18116 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:09.315724 18116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:09.322867 18116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Generated new TSK 0
I20260620 14:33:09.323793 18116 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:33:09.356195 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:09.471402 18126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:09.471715 18126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:09.471760 18126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:09.475939 18126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:09.476014 18126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:09.476080 18126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:09.481678 18126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:09.482811 18126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:09.483852 18126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:09.493587 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.493587 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.493677 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:09.494019 18126 server_base.cc:1034] running on GCE node
I20260620 14:33:09.494544 18126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:09.495221 18126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:09.496392 18126 hybrid_clock.cc:648] HybridClock initialized: now 1781965989496369 us; error 42 us; skew 500 ppm
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:09.499312 18126 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:09.500345 18126 webserver.cc:460] Webserver started at http://127.4.58.65:46355/ using document root <none> and password file <none>
I20260620 14:33:09.500976 18126 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:09.501049 18126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:09.501264 18126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:09.503363 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "db61d9fb63554ee7a875f7db0849df20"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "1e71fdef2c276f3d88aecc1a4b549149"
server_key_iv: "bcbe403a094b8620cb4df80b81c63741"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.503931 18126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "db61d9fb63554ee7a875f7db0849df20"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "1e71fdef2c276f3d88aecc1a4b549149"
server_key_iv: "bcbe403a094b8620cb4df80b81c63741"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.507968 18126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:33:09.510749 18141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.512195 18126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:09.512331 18126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "db61d9fb63554ee7a875f7db0849df20"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "1e71fdef2c276f3d88aecc1a4b549149"
server_key_iv: "bcbe403a094b8620cb4df80b81c63741"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.512445 18126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:09.531134 18126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:09.534767 18126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:09.534951 18126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:09.535652 18126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:09.536584 18126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:09.536631 18126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.536700 18126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:09.536728 18126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.545620 18126 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:35357
I20260620 14:33:09.545656 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:35357 every 8 connection(s)
I20260620 14:33:09.546553 18126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:09.553512 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18126
I20260620 14:33:09.553642 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:33:09.553980 4329 external_mini_cluster.cc:1428] Setting key 345bd7c5060d4517a284e630617ebb63
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:09.560619 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.548490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:50923 (local address 127.4.58.126:36951)
0620 14:33:09.548646 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:09.548649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:09.549723 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 14:33:09.550658 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:09.550662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:09.550720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:09.550802 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:09.552054 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.552604 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.553886 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.554179 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.554277 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:09.557018 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:09.557021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:09.557023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:09.557048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:09.558421 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.558954 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.558957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.558959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.559036 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.559395 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.559398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.559399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.559594 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:09.559699 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:09.560284 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 14:33:09.560433 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:33:09.561578 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.548853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:50923)
0620 14:33:09.549528 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:09.549574 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:09.550421 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:09.550998 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:09.551005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:09.551377 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:09.551881 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:09.551895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.552739 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.552742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:09.553752 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:09.553763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.554337 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.554341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:09.554509 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:09.555098 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:09.555116 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:09.556849 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:09.558542 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:09.558548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:09.558566 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:09.558841 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:09.559130 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:09.559133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:09.559135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:09.559290 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:09.559719 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:09.559726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:09.560012 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 14:33:09.560211 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 14:33:09.560593 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":123,"threads_started":1}
I20260620 14:33:09.563113 18255 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:33:09.563374 18255 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:09.563915 18255 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:09.565667 18064 ts_manager.cc:194] Registered new tserver with Master: db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:33:09.566890 18064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:50923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:33:09.583988 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:09.696112 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:09.696399 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:09.696444 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:09.700673 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:09.700740 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:09.700838 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:09.706288 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:09.707362 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:09.708402 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:09.718623 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:09.718778 18260 server_base.cc:1034] running on GCE node
W20260620 14:33:09.718626 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.718623 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:09.719522 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:09.720268 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:09.721429 18260 hybrid_clock.cc:648] HybridClock initialized: now 1781965989721384 us; error 64 us; skew 500 ppm
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:09.724164 18260 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:09.725142 18260 webserver.cc:460] Webserver started at http://127.4.58.66:46839/ using document root <none> and password file <none>
I20260620 14:33:09.725726 18260 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:09.725800 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:09.726004 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:09.728134 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "0a65a6b2f6884f049a7080a796ccfba9"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "2695a1a1bf3ebf6fbc140a0d3d50cbdd"
server_key_iv: "cecd2c7ef1123967c82a87afa6eeddeb"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.728705 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "0a65a6b2f6884f049a7080a796ccfba9"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "2695a1a1bf3ebf6fbc140a0d3d50cbdd"
server_key_iv: "cecd2c7ef1123967c82a87afa6eeddeb"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.732512 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:33:09.735067 18275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.736433 18260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:09.736552 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0a65a6b2f6884f049a7080a796ccfba9"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "2695a1a1bf3ebf6fbc140a0d3d50cbdd"
server_key_iv: "cecd2c7ef1123967c82a87afa6eeddeb"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.736682 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:09.755638 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:09.758992 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:09.759166 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:09.759799 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:09.760668 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:09.760717 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.760797 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:09.760846 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.769843 18260 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:45839
I20260620 14:33:09.769876 18388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:45839 every 8 connection(s)
I20260620 14:33:09.770804 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:09.780822 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18260
I20260620 14:33:09.780925 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:33:09.781232 4329 external_mini_cluster.cc:1428] Setting key 0cbf8b8b95149545963e2027177ae1f7
I20260620 14:33:09.785121 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.772706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:36411 (local address 127.4.58.126:36951)
0620 14:33:09.772850 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:09.772854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:09.774031 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 14:33:09.774913 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:09.774916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:09.774968 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:09.775044 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:09.776237 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.776855 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.778041 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.778287 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.778394 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:09.781085 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:09.781089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:09.781092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:09.781123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:09.782869 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.783459 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.783463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.783465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.783565 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:09.783912 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:09.783915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:09.783916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:09.784154 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:09.784247 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:09.784805 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 14:33:09.784942 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:33:09.786108 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.773169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:36411)
0620 14:33:09.773874 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:09.773919 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:09.774716 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:09.775189 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:09.775196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:09.775565 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:09.776083 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:09.776099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.776982 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.776985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:09.777939 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:09.777949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:09.778431 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:09.778434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:09.778588 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:09.779173 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:09.779190 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:09.780862 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:09.783009 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:09.783017 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:09.783034 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:09.783355 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:09.783682 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:09.783684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:09.783685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:09.783829 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:09.784248 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:09.784254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:09.784515 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 14:33:09.784701 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:33:09.785083 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":121,"threads_started":1}
I20260620 14:33:09.787439 18389 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
I20260620 14:33:09.787717 18389 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:09.788259 18389 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:09.789535 18064 ts_manager.cc:194] Registered new tserver with Master: 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839)
I20260620 14:33:09.790344 18064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:36411
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:33:09.809646 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:09.922267 18394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:09.922570 18394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:09.922646 18394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:09.926995 18394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:09.927075 18394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:09.927160 18394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:09.932693 18394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:09.933897 18394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:09.934954 18394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:09.944545 18400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.944556 18399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:09.944556 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:09.944856 18394 server_base.cc:1034] running on GCE node
I20260620 14:33:09.945250 18394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:09.945927 18394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:09.947094 18394 hybrid_clock.cc:648] HybridClock initialized: now 1781965989947074 us; error 38 us; skew 500 ppm
Jun 20 14:33:09 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:09.949934 18394 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:09.950855 18394 webserver.cc:460] Webserver started at http://127.4.58.67:40849/ using document root <none> and password file <none>
I20260620 14:33:09.951401 18394 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:09.951467 18394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:09.951685 18394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:09.953815 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "a6b4e050ec4e490eb58fe2f575432300"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "6d8f668f19fd5560df4efd2fefee68a4"
server_key_iv: "963dc48e5a1a3445e5753b94e426679e"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.954358 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "a6b4e050ec4e490eb58fe2f575432300"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "6d8f668f19fd5560df4efd2fefee68a4"
server_key_iv: "963dc48e5a1a3445e5753b94e426679e"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.958114 18394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:33:09.960587 18409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.961894 18394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:33:09.962013 18394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "a6b4e050ec4e490eb58fe2f575432300"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "6d8f668f19fd5560df4efd2fefee68a4"
server_key_iv: "963dc48e5a1a3445e5753b94e426679e"
server_key_version: "encryptionkey@0"
I20260620 14:33:09.962121 18394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:09.982555 18394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:09.985570 18394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:09.985770 18394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:09.986415 18394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:09.987277 18394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:09.987327 18394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.987381 18394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:09.987411 18394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:09.996251 18394 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:38873
I20260620 14:33:09.996284 18522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:38873 every 8 connection(s)
I20260620 14:33:09.997264 18394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:10.006296 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18394
I20260620 14:33:10.006458 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:33:10.006783 4329 external_mini_cluster.cc:1428] Setting key 47a54ca533d77f4af564d705c5c4428e
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965989, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:10.011888 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.999153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:59307 (local address 127.4.58.126:36951)
0620 14:33:09.999284 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:09.999287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.000340 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 14:33:10.001336 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.001340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.001401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.001475 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.002759 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.003311 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.004538 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.004810 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.004911 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.007785 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:10.007788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:10.007790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:10.007818 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:10.009500 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.010151 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.010154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.010155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.010245 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.010679 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.010682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.010684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.010898 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:10.010986 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.011596 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.011747 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:33:10.012813 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:09.999493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:59307)
0620 14:33:10.000171 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.000214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.001101 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.001677 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.001685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.002056 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:10.002590 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.002611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.003451 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.003454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.004388 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.004399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.004970 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.004974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.005156 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.005771 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:10.005789 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:10.007587 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:10.009661 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.009667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.009684 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.009997 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.010341 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.010343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.010345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.010516 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.011015 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:10.011021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:10.011318 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.011510 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.011841 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":118,"threads_started":1}
I20260620 14:33:10.014264 18523 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
I20260620 14:33:10.014578 18523 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:10.015148 18523 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:10.016376 18064 ts_manager.cc:194] Registered new tserver with Master: a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
I20260620 14:33:10.017071 18064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:59307
I20260620 14:33:10.021209 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:10.031279 18103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.023233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.4.58.126:36951)
0620 14:33:10.023353 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.023357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.023518 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:33:10.023645 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.023647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.023690 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.023748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.024439 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.025039 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.025936 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.026195 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.026281 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.028019 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:10.028023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:10.028026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:10.028058 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:10.029634 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.030111 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.030115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.030116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.030206 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.030420 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.030422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.030423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.030679 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:10.030785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.030976 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.031114 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:33:10.034431 18064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:10.037595 18064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:10.053035 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.047184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.0.0.1:58122)
0620 14:33:10.047648 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.047660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.047737 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.048439 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.048443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.048460 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:10.048558 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.048564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.050013 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.050017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.051160 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.051171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052364 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.052530 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.052584 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.052670 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.052864 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":18,"thread_start_us":106,"threads_started":1}
I20260620 14:33:10.053045 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.047344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.4.58.66:45839)
0620 14:33:10.048140 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.048152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.048177 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:33:10.048256 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.048260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.048439 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.048580 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:10.048918 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.049672 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.051652 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052222 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052324 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.052503 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.052741 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.052883 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":112,"threads_started":1}
I20260620 14:33:10.053092 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.046593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.4.58.67:38873)
0620 14:33:10.047502 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.047515 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.047540 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:33:10.047623 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.047629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.047860 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.048030 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:10.048532 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.049796 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.051418 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052223 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052331 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.052508 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.052833 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.052936 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":114,"threads_started":1}
I20260620 14:33:10.053377 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.046589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.0.0.1:36102)
0620 14:33:10.047000 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.047055 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.047200 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.048697 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.048700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.048716 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:10.048817 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.048822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.049808 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.049811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.051539 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.051547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052349 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.052529 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.052581 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.052687 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.053263 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":610,"spinlock_wait_cycles":1664,"thread_start_us":121,"threads_started":1}
I20260620 14:33:10.053426 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.047711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.4.58.65:35357)
0620 14:33:10.048013 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.048027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.048048 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:33:10.048116 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.048121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.048297 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.048445 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:10.048674 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.049894 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.051307 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052223 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052962 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.053127 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.053181 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.053274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260620 14:33:10.053547 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.046452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:38873 (local address 127.0.0.1:49796)
0620 14:33:10.047001 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.047054 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.047199 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.048223 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.048229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.048259 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:10.048407 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.048414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.049940 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.049943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.051290 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.051300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.052569 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.052573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.052651 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.052670 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.052724 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.052863 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":119,"spinlock_wait_cycles":7296,"thread_start_us":88,"threads_started":1}
I20260620 14:33:10.056495 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 62a31481849e4061a807171bfa459b25 (DEFAULT_TABLE table=test-table [id=51a476af18ac4c9d94e7a61363a62322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:10.056548 18189 tablet_service.cc:1467] Processing CreateTablet for tablet 62a31481849e4061a807171bfa459b25 (DEFAULT_TABLE table=test-table [id=51a476af18ac4c9d94e7a61363a62322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:10.057585 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a31481849e4061a807171bfa459b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:10.057693 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a31481849e4061a807171bfa459b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:10.057809 18323 tablet_service.cc:1467] Processing CreateTablet for tablet 62a31481849e4061a807171bfa459b25 (DEFAULT_TABLE table=test-table [id=51a476af18ac4c9d94e7a61363a62322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:10.058887 18323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a31481849e4061a807171bfa459b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:10.064375 18540 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Bootstrap starting.
I20260620 14:33:10.066335 18542 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Bootstrap starting.
I20260620 14:33:10.066900 18541 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Bootstrap starting.
I20260620 14:33:10.067305 18540 tablet_bootstrap.cc:654] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:10.068059 18540 log.cc:824] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:10.069160 18542 tablet_bootstrap.cc:654] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:10.069180 18541 tablet_bootstrap.cc:654] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:10.070062 18542 log.cc:824] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:10.070063 18541 log.cc:824] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:10.070726 18540 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: No bootstrap required, opened a new log
I20260620 14:33:10.070964 18540 ts_tablet_manager.cc:1397] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:33:10.072227 18542 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: No bootstrap required, opened a new log
I20260620 14:33:10.072436 18542 ts_tablet_manager.cc:1397] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:33:10.072512 18541 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: No bootstrap required, opened a new log
I20260620 14:33:10.072773 18541 ts_tablet_manager.cc:1397] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:33:10.074189 18540 raft_consensus.cc:348] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.074445 18540 raft_consensus.cc:374] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:10.074519 18540 raft_consensus.cc:729] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b4e050ec4e490eb58fe2f575432300, State: Initialized, Role: FOLLOWER
I20260620 14:33:10.075004 18540 consensus_queue.cc:260] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.075696 18523 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
I20260620 14:33:10.076077 18540 ts_tablet_manager.cc:1428] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:33:10.076407 18542 raft_consensus.cc:348] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.076825 18542 raft_consensus.cc:374] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:10.076929 18542 raft_consensus.cc:729] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a65a6b2f6884f049a7080a796ccfba9, State: Initialized, Role: FOLLOWER
I20260620 14:33:10.077463 18542 consensus_queue.cc:260] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.078177 18541 raft_consensus.cc:348] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.078410 18541 raft_consensus.cc:374] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:10.078491 18541 raft_consensus.cc:729] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db61d9fb63554ee7a875f7db0849df20, State: Initialized, Role: FOLLOWER
I20260620 14:33:10.078606 18542 ts_tablet_manager.cc:1428] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:33:10.078955 18541 consensus_queue.cc:260] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.079095 18389 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
I20260620 14:33:10.079730 18255 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
I20260620 14:33:10.079973 18541 ts_tablet_manager.cc:1428] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:33:10.103420 18546 raft_consensus.cc:482] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:10.103658 18546 raft_consensus.cc:504] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.104873 18546 leader_election.cc:288] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
I20260620 14:33:10.109189 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.105219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:48967 (local address 127.4.58.66:45839)
0620 14:33:10.105359 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.105364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.105380 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:33:10.105534 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.105537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.105582 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.105649 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:10.105915 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.106549 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.107926 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.108549 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.108666 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.108735 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.108989 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.109066 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:33:10.109200 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.105119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.4.58.67:48967)
0620 14:33:10.105284 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.105301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.105404 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.105672 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.105675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.105694 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:10.105812 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.105818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.106680 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.106683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.107799 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.107809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.108714 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.108727 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.108857 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.108878 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.108945 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.109029 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":21}
I20260620 14:33:10.109793 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.105128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.4.58.67:46011)
0620 14:33:10.105690 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.105702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.105795 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.106133 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.106137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.106154 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:10.106268 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.106274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.107417 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.107420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.108535 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.108547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.109410 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.109414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.109511 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.109533 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.109605 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.109673 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":213,"thread_start_us":132,"threads_started":1}
I20260620 14:33:10.109855 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.105255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46011 (local address 127.4.58.65:35357)
0620 14:33:10.105400 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.105405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.105771 (+ 366us) server_negotiation.cc:407] Connection header received
0620 14:33:10.105896 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.105899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.106017 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.106079 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:10.106524 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.107296 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.108679 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.109268 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.109375 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.109441 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.109645 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.109732 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:33:10.109982 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a31481849e4061a807171bfa459b25" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a65a6b2f6884f049a7080a796ccfba9" is_pre_election: true
I20260620 14:33:10.110325 18343 raft_consensus.cc:2454] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 0.
I20260620 14:33:10.110415 18209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a31481849e4061a807171bfa459b25" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db61d9fb63554ee7a875f7db0849df20" is_pre_election: true
I20260620 14:33:10.110725 18209 raft_consensus.cc:2454] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 0.
I20260620 14:33:10.110838 18411 leader_election.cc:302] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300; no voters:
I20260620 14:33:10.111159 18546 raft_consensus.cc:2781] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:10.111250 18546 raft_consensus.cc:482] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:10.111295 18546 raft_consensus.cc:3037] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:10.112349 18546 raft_consensus.cc:504] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.112707 18546 leader_election.cc:288] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 election: Requested vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
I20260620 14:33:10.113185 18209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a31481849e4061a807171bfa459b25" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db61d9fb63554ee7a875f7db0849df20"
I20260620 14:33:10.113185 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a31481849e4061a807171bfa459b25" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a65a6b2f6884f049a7080a796ccfba9"
I20260620 14:33:10.113332 18209 raft_consensus.cc:3037] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:10.113332 18343 raft_consensus.cc:3037] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:10.114691 18209 raft_consensus.cc:2454] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 1.
I20260620 14:33:10.114691 18343 raft_consensus.cc:2454] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 1.
I20260620 14:33:10.115042 18411 leader_election.cc:302] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300; no voters:
I20260620 14:33:10.115264 18546 raft_consensus.cc:2781] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:10.115510 18546 raft_consensus.cc:686] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 1 LEADER]: Becoming Leader. State: Replica: a6b4e050ec4e490eb58fe2f575432300, State: Running, Role: LEADER
I20260620 14:33:10.115900 18546 consensus_queue.cc:237] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.119421 18063 catalog_manager.cc:5495] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "a6b4e050ec4e490eb58fe2f575432300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } health_report { overall_health: UNKNOWN } } }
I20260620 14:33:10.133612 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.129766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.4.58.67:38873)
0620 14:33:10.129882 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.129885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.129899 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:33:10.129979 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.129981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.130041 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.130139 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:10.130412 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.130953 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.131911 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.132147 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.132239 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.132628 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:10.133188 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:10.133301 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.133416 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.133481 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:33:10.140188 18209 raft_consensus.cc:1261] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Refusing update from remote peer a6b4e050ec4e490eb58fe2f575432300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:10.140208 18343 raft_consensus.cc:1261] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Refusing update from remote peer a6b4e050ec4e490eb58fe2f575432300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:10.141039 18550 consensus_queue.cc:1035] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:10.141248 18546 consensus_queue.cc:1035] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:10.150091 18554 mvcc.cc:205] Tried to move back new op lower bound from 7298932695606321152 to 7298932695517868032. Current Snapshot: MvccSnapshot[applied={T|T < 7298932695606321152}]
I20260620 14:33:10.159162 18063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:10.159346 18063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:10.162076 18063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Sending DeleteTablet for 3 replicas of tablet 62a31481849e4061a807171bfa459b25
I20260620 14:33:10.163144 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18126
I20260620 14:33:10.163125 18323 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a31481849e4061a807171bfa459b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:10 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:36102
I20260620 14:33:10.163260 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a31481849e4061a807171bfa459b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:10 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:49796
I20260620 14:33:10.164289 18563 tablet_replica.cc:330] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: stopping tablet replica
I20260620 14:33:10.164572 18563 raft_consensus.cc:2229] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:10.164906 18563 raft_consensus.cc:2258] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:10.165299 18564 tablet_replica.cc:330] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: stopping tablet replica
I20260620 14:33:10.165623 18564 raft_consensus.cc:2229] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:10.166007 18564 raft_consensus.cc:2258] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:10.167711 18563 ts_tablet_manager.cc:1905] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:10.167711 18564 ts_tablet_manager.cc:1905] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:10.170666 18564 ts_tablet_manager.cc:1918] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:10.170698 18563 ts_tablet_manager.cc:1918] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:10.170786 18563 log.cc:1196] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/62a31481849e4061a807171bfa459b25
I20260620 14:33:10.170941 18564 log.cc:1196] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/62a31481849e4061a807171bfa459b25
W20260620 14:33:10.171021 18051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:35357 (error 108)
I20260620 14:33:10.171147 18563 ts_tablet_manager.cc:1939] T 62a31481849e4061a807171bfa459b25 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting consensus metadata
I20260620 14:33:10.171325 18564 ts_tablet_manager.cc:1939] T 62a31481849e4061a807171bfa459b25 P a6b4e050ec4e490eb58fe2f575432300: Deleting consensus metadata
I20260620 14:33:10.171991 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18260
I20260620 14:33:10.173756 18051 catalog_manager.cc:4841] TS a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): tablet 62a31481849e4061a807171bfa459b25 (table test-table [id=51a476af18ac4c9d94e7a61363a62322]) successfully deleted
I20260620 14:33:10.173882 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.173579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.0.0.1:58132)
0620 14:33:10.173719 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.173800 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:33:10.174042 18051 catalog_manager.cc:4584] TS db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62a31481849e4061a807171bfa459b25: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35357: connect: Connection refused (error 111)
I20260620 14:33:10.177791 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18394
I20260620 14:33:10.178536 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.177958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.0.0.1:36104)
0620 14:33:10.178386 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.178445 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
I20260620 14:33:10.183154 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18032
I20260620 14:33:10.189291 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--webserver_interface=127.4.58.126
--webserver_port=46561
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:10.303633 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:10.303937 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:10.303982 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:10.308235 18566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:10.308322 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:10.308341 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:10.308357 18566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:10.308374 18566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:10.313980 18566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:36951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:10.315135 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:10.316159 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:10.329615 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.329676 18566 server_base.cc:1034] running on GCE node
W20260620 14:33:10.329782 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.329615 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.330734 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:10.331998 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:10.333182 18566 hybrid_clock.cc:648] HybridClock initialized: now 1781965990333167 us; error 32 us; skew 500 ppm
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:10.340821 18566 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:10.341852 18566 webserver.cc:460] Webserver started at http://127.4.58.126:46561/ using document root <none> and password file <none>
I20260620 14:33:10.342621 18566 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:10.342734 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:10.347163 18566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:33:10.349632 18582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.351029 18566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:10.351217 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e754f05539bc43148b4926445964406f"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "bd3818b9160dcecac16f1d2c4a51a42c"
server_key_iv: "e08b87b53c78d9d065a12e200c146afa"
server_key_version: "encryptionkey@0"
I20260620 14:33:10.351866 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:10.374266 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:10.378327 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:10.378528 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:10.387235 18566 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:36951
I20260620 14:33:10.387513 18634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:36951 every 8 connection(s)
I20260620 14:33:10.388816 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:10.394968 18635 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:10.396513 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18566
I20260620 14:33:10.397091 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:35357
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=46355
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:10.397359 18635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap starting.
I20260620 14:33:10.420934 18635 log.cc:824] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:10.430374 18635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:10.430745 18635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap complete.
I20260620 14:33:10.434785 18635 raft_consensus.cc:348] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:10.435230 18635 raft_consensus.cc:729] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e754f05539bc43148b4926445964406f, State: Initialized, Role: FOLLOWER
I20260620 14:33:10.435717 18635 consensus_queue.cc:260] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:10.435842 18635 raft_consensus.cc:388] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:10.435917 18635 raft_consensus.cc:482] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:10.436017 18635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:33:10.438323 18635 raft_consensus.cc:504] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:10.438679 18635 leader_election.cc:302] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e754f05539bc43148b4926445964406f; no voters:
I20260620 14:33:10.439023 18635 leader_election.cc:288] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:33:10.439128 18639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:33:10.439327 18639 raft_consensus.cc:686] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 LEADER]: Becoming Leader. State: Replica: e754f05539bc43148b4926445964406f, State: Running, Role: LEADER
I20260620 14:33:10.439734 18639 consensus_queue.cc:237] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:10.440328 18635 sys_catalog.cc:564] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:10.441256 18640 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:10.441394 18640 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:10.441787 18641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e754f05539bc43148b4926445964406f. Latest consensus state: current_term: 2 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:10.441869 18648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:10.441879 18641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:10.446810 18648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:10.447669 18648 catalog_manager.cc:1259] Loaded cluster ID: b1d9343b43ad4f8f809b5da1e976a06f
I20260620 14:33:10.447748 18648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:10.448930 18648 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:33:10.451680 18648 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:33:10.456490 18648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:10.457254 18648 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Loaded TSK: 0
I20260620 14:33:10.458252 18648 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:33:10.531178 18637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:10.531483 18637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:10.531548 18637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:10.535827 18637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:10.535903 18637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:10.535989 18637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:10.541702 18637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:35357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=46355
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:10.542845 18637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:10.543898 18637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:10.553826 18662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.553846 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.553846 18665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.554430 18637 server_base.cc:1034] running on GCE node
I20260620 14:33:10.554917 18637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:10.555783 18637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:10.556974 18637 hybrid_clock.cc:648] HybridClock initialized: now 1781965990556952 us; error 36 us; skew 500 ppm
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:10.560566 18637 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:10.561602 18637 webserver.cc:460] Webserver started at http://127.4.58.65:46355/ using document root <none> and password file <none>
I20260620 14:33:10.562168 18637 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:10.562244 18637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:10.566461 18637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:33:10.569036 18672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.570497 18637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:10.570648 18637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "db61d9fb63554ee7a875f7db0849df20"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "1e71fdef2c276f3d88aecc1a4b549149"
server_key_iv: "bcbe403a094b8620cb4df80b81c63741"
server_key_version: "encryptionkey@0"
I20260620 14:33:10.571252 18637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:10.589690 18637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:10.592900 18637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:10.593087 18637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:10.593709 18637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:10.594910 18679 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:10.599875 18637 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:10.599982 18637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 14:33:10.600059 18637 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:10.602000 18637 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:10.602059 18637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 14:33:10.602380 18679 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Bootstrap starting.
I20260620 14:33:10.611547 18637 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:35357
I20260620 14:33:10.611660 18786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:35357 every 8 connection(s)
I20260620 14:33:10.612603 18637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:10.615159 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18637
I20260620 14:33:10.615684 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:45839
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=46839
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:10.622263 18679 log.cc:824] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:10.631266 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.614625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:43499)
0620 14:33:10.615535 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.615591 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.616499 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.618178 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.618185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.618530 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:10.619000 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.619015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.620368 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.620371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.621307 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.621317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.622088 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.622093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.622285 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.622960 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:10.622978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:10.624946 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:10.627571 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.627578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.627593 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.627964 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.628406 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.628409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.628411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.628635 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.629400 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:10.629407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:10.629730 (+ 323us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.629933 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.630280 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":147,"threads_started":1}
I20260620 14:33:10.631268 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.614753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:43499 (local address 127.4.58.126:36951)
0620 14:33:10.615230 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.615252 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.615735 (+ 483us) server_negotiation.cc:407] Connection header received
0620 14:33:10.616993 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.617010 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.617394 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.617721 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.619195 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.620196 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.621437 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.621929 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.622218 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.625218 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:10.625225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:10.625245 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:10.625283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:10.627419 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.628111 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.628117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.628123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.628284 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.628745 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.628781 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.628783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.629268 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:10.629406 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.630084 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.630464 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
I20260620 14:33:10.632349 18679 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:10.632769 18679 tablet_bootstrap.cc:492] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Bootstrap complete.
I20260620 14:33:10.634037 18679 ts_tablet_manager.cc:1397] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.009s
I20260620 14:33:10.634804 18787 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
I20260620 14:33:10.635051 18787 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:10.635643 18787 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:10.636948 18679 raft_consensus.cc:348] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.637452 18679 raft_consensus.cc:729] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db61d9fb63554ee7a875f7db0849df20, State: Initialized, Role: FOLLOWER
I20260620 14:33:10.638032 18679 consensus_queue.cc:260] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:10.638640 18596 ts_manager.cc:194] Registered new tserver with Master: db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
I20260620 14:33:10.639209 18679 ts_tablet_manager.cc:1428] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:33:10.642499 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:43499
I20260620 14:33:10.645251 18787 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:33:10.653226 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.641830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.0.0.1:58142)
0620 14:33:10.643423 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.643456 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.643588 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.644058 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.644064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.644475 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:10.644580 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.644586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.645602 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.645606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.646530 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.646541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.647289 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.647293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.647400 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.647938 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:10.647950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:10.649532 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:10.651371 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.651381 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.651387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.651620 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.651916 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.651920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.651922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.652003 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.652482 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:10.652488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:10.652632 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.652788 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.653038 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"thread_start_us":102,"threads_started":1}
I20260620 14:33:10.653226 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.641908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.4.58.65:35357)
0620 14:33:10.642312 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.642322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.643551 (+ 1229us) server_negotiation.cc:407] Connection header received
0620 14:33:10.643737 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.643741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.643905 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.644027 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.644702 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.645489 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.646675 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.647169 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.647267 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.649684 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:10.649696 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:10.649707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:10.649731 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:10.651250 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.651736 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.651743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.651744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.651826 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.652091 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.652094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.652096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.652365 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:10.652485 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.652890 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.653037 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":103,"threads_started":1}
I20260620 14:33:10.654220 18721 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a31481849e4061a807171bfa459b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:10 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:58142
I20260620 14:33:10.655141 18796 tablet_replica.cc:330] stopping tablet replica
I20260620 14:33:10.655483 18796 raft_consensus.cc:2229] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:10.655757 18796 raft_consensus.cc:2258] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:10.656904 18796 ts_tablet_manager.cc:1905] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:10.661783 18796 ts_tablet_manager.cc:1918] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:10.661907 18796 log.cc:1196] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/62a31481849e4061a807171bfa459b25
I20260620 14:33:10.662158 18796 ts_tablet_manager.cc:1939] T 62a31481849e4061a807171bfa459b25 P db61d9fb63554ee7a875f7db0849df20: Deleting consensus metadata
I20260620 14:33:10.664460 18586 catalog_manager.cc:4841] TS db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357): tablet 62a31481849e4061a807171bfa459b25 (table test-table [id=51a476af18ac4c9d94e7a61363a62322]) successfully deleted
W20260620 14:33:10.739831 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:10.740126 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:10.740173 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:10.744429 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:10.744508 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:10.744570 18791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:10.750346 18791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=46839
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:10.751462 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:10.752525 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:10.762841 18804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.762915 18791 server_base.cc:1034] running on GCE node
W20260620 14:33:10.762857 18802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.762854 18801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.763494 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:10.764245 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:10.765414 18791 hybrid_clock.cc:648] HybridClock initialized: now 1781965990765380 us; error 46 us; skew 500 ppm
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:10.768280 18791 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:10.769265 18791 webserver.cc:460] Webserver started at http://127.4.58.66:46839/ using document root <none> and password file <none>
I20260620 14:33:10.769825 18791 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:10.769899 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:10.774027 18791 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:33:10.776398 18811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.778038 18791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:33:10.778169 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0a65a6b2f6884f049a7080a796ccfba9"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "2695a1a1bf3ebf6fbc140a0d3d50cbdd"
server_key_iv: "cecd2c7ef1123967c82a87afa6eeddeb"
server_key_version: "encryptionkey@0"
I20260620 14:33:10.778779 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:10.793519 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:10.796864 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:10.797061 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:10.797725 18791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:10.798585 18791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:10.798631 18791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.798676 18791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:10.798704 18791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.807368 18791 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:45839
I20260620 14:33:10.807386 18924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:45839 every 8 connection(s)
I20260620 14:33:10.808405 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:10.815860 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18791
I20260620 14:33:10.816255 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:38873
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=40849
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:10.822310 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.810316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:57019 (local address 127.4.58.126:36951)
0620 14:33:10.810490 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.810493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.811444 (+ 951us) server_negotiation.cc:407] Connection header received
0620 14:33:10.812339 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.812343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.812405 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.812481 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.813768 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.814306 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.815450 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.815725 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.815817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.818552 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:10.818556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:10.818559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:10.818595 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:10.820032 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.820628 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.820631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.820634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.820719 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:10.821090 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:10.821094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:10.821096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:10.821303 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:10.821392 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:10.821980 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 14:33:10.822138 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:33:10.823225 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.810614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:57019)
0620 14:33:10.811271 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.811315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.812123 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.812700 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.812708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.813123 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:10.813601 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.813626 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.814434 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.814437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.815322 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.815331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:10.815874 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:10.815878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:10.816059 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:10.816689 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:10.816708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:10.818365 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:10.820191 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.820198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.820215 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.820520 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.820837 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:10.820840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:10.820843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:10.820993 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:10.821410 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:10.821417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:10.821737 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 14:33:10.821915 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:33:10.822277 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":102,"threads_started":1}
I20260620 14:33:10.824563 18925 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
I20260620 14:33:10.824836 18925 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:10.825338 18925 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:10.826493 18596 ts_manager.cc:194] Registered new tserver with Master: 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839)
I20260620 14:33:10.827075 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:57019
W20260620 14:33:10.930233 18928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:10.930516 18928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:10.930577 18928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:10.934728 18928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:10.934803 18928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:10.934885 18928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:10.940474 18928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:38873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=40849
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:10.941749 18928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:10.942883 18928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:10.953040 18934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.953266 18936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:10.953027 18933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:10.953348 18928 server_base.cc:1034] running on GCE node
I20260620 14:33:10.953775 18928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:10.954444 18928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:10.955621 18928 hybrid_clock.cc:648] HybridClock initialized: now 1781965990955594 us; error 44 us; skew 500 ppm
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:10.958729 18928 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:10.959774 18928 webserver.cc:460] Webserver started at http://127.4.58.67:40849/ using document root <none> and password file <none>
I20260620 14:33:10.960343 18928 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:10.960427 18928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:10.964708 18928 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:33:10.966943 18943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.968289 18928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:10.968407 18928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "a6b4e050ec4e490eb58fe2f575432300"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "6d8f668f19fd5560df4efd2fefee68a4"
server_key_iv: "963dc48e5a1a3445e5753b94e426679e"
server_key_version: "encryptionkey@0"
I20260620 14:33:10.969022 18928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:10.979560 18928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:10.982698 18928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:10.982882 18928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:10.983480 18928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:10.984290 18928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:10.984346 18928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.984400 18928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:10.984429 18928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:10.993197 18928 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:38873
I20260620 14:33:10.993232 19056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:38873 every 8 connection(s)
I20260620 14:33:10.994227 18928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:11.003397 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 18928
Jun 20 14:33:10 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965990, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:11.008126 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.996038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:34981 (local address 127.4.58.126:36951)
0620 14:33:10.996190 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:10.996194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:10.997265 (+ 1071us) server_negotiation.cc:407] Connection header received
0620 14:33:10.998167 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:10.998170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:10.998222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:10.998298 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:10.999483 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.000019 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.001264 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.001494 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.001588 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.004318 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:11.004323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:11.004325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:11.004356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:11.005780 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:11.006441 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:11.006444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:11.006445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:11.006527 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:11.006905 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:11.006909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:11.006910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:11.007133 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:11.007236 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.007846 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.007959 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:33:11.009305 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:10.996406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:34981)
0620 14:33:10.997102 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:10.997154 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:10.997955 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:10.998466 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:10.998473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:10.998852 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:10.999350 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:10.999364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.000144 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.000147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.001131 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.001140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.001643 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.001646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.001789 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.002385 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:11.002404 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:11.004108 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:11.005900 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:11.005907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:11.005935 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:11.006335 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:11.006628 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:11.006631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:11.006633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:11.006810 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:11.007249 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:11.007255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:11.007544 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 14:33:11.007797 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 14:33:11.008299 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":121,"threads_started":1}
I20260620 14:33:11.010779 19057 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36951
I20260620 14:33:11.011034 19057 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:11.011561 19057 heartbeater.cc:511] Master 127.4.58.126:36951 requested a full tablet report, sending...
I20260620 14:33:11.012692 18596 ts_manager.cc:194] Registered new tserver with Master: a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
I20260620 14:33:11.012959 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.004154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.4.58.126:36951)
0620 14:33:11.004473 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.004478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.004636 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:33:11.004811 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.004814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.004859 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.004939 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:11.005752 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.006530 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.007601 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.007935 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.008045 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.009511 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:11.009515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:11.009518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:11.009546 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:11.011247 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:11.011657 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:11.011661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:11.011663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:11.011752 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:11.012009 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:11.012012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:11.012014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:11.012242 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:11.012417 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.012656 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.012816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260620 14:33:11.013340 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:34981
I20260620 14:33:11.015306 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:11.019443 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.015906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.4.58.126:36951)
0620 14:33:11.016023 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.016028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.016198 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:33:11.016321 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.016324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.016424 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.016505 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:11.016688 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.017235 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.018052 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.018287 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.018370 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.018688 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:11.019072 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:11.019171 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.019263 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.019318 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:33:11.020085 18596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:11.023419 18596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:11.032876 18721 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7b44a33bae4dbfb29799ad25a9d557 (DEFAULT_TABLE table=test-table [id=73015a47a4a44b1b84d8455ba6a59de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:11.033556 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7b44a33bae4dbfb29799ad25a9d557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:11.035571 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.030064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.0.0.1:36106)
0620 14:33:11.030226 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.030240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:11.030319 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:11.031528 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:11.031531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:11.031551 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:11.031673 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.031679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.033138 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.033141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.034281 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.034291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.035134 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.035138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.035232 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.035278 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:33:11.035350 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:33:11.035436 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:33:11.035614 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.030381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.4.58.66:45839)
0620 14:33:11.030898 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.030913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.030939 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:33:11.031076 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.031085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.031363 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.031543 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:11.031773 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.032976 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.034412 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.034995 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.035096 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.035256 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.035382 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.035476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":120,"threads_started":1}
I20260620 14:33:11.036619 18679 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Bootstrap starting.
I20260620 14:33:11.036813 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.030767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:38873 (local address 127.0.0.1:49814)
0620 14:33:11.031230 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.031244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:11.031333 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:11.032075 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:11.032080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:11.032097 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:11.032213 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.032242 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.033873 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.033876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.034947 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.034958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.036374 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.036378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.036466 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.036490 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:33:11.036558 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:33:11.036645 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":93,"threads_started":1}
I20260620 14:33:11.037214 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.031077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.4.58.67:38873)
0620 14:33:11.031562 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.031574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.031595 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:33:11.031665 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.031670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.031883 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.032034 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:11.032363 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.033728 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.035090 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.036050 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.036163 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.036307 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.036963 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.037072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":156,"threads_started":1}
I20260620 14:33:11.037537 18679 tablet_bootstrap.cc:654] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:11.039152 18679 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: No bootstrap required, opened a new log
I20260620 14:33:11.039243 18679 ts_tablet_manager.cc:1397] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:11.038961 18859 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7b44a33bae4dbfb29799ad25a9d557 (DEFAULT_TABLE table=test-table [id=73015a47a4a44b1b84d8455ba6a59de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:11.039742 18679 raft_consensus.cc:348] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.039862 18679 raft_consensus.cc:374] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:11.039897 18679 raft_consensus.cc:729] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db61d9fb63554ee7a875f7db0849df20, State: Initialized, Role: FOLLOWER
I20260620 14:33:11.039788 18991 tablet_service.cc:1467] Processing CreateTablet for tablet 4f7b44a33bae4dbfb29799ad25a9d557 (DEFAULT_TABLE table=test-table [id=73015a47a4a44b1b84d8455ba6a59de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:11.040189 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7b44a33bae4dbfb29799ad25a9d557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:11.040249 18679 consensus_queue.cc:260] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.040825 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f7b44a33bae4dbfb29799ad25a9d557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:11.041385 18679 ts_tablet_manager.cc:1428] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:33:11.047660 19071 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Bootstrap starting.
I20260620 14:33:11.047660 19070 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Bootstrap starting.
I20260620 14:33:11.049950 19071 tablet_bootstrap.cc:654] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:11.050151 19070 tablet_bootstrap.cc:654] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:11.050882 19070 log.cc:824] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:11.050930 19071 log.cc:824] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:11.052644 19070 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: No bootstrap required, opened a new log
I20260620 14:33:11.052659 19071 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: No bootstrap required, opened a new log
I20260620 14:33:11.052847 19071 ts_tablet_manager.cc:1397] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:33:11.052851 19070 ts_tablet_manager.cc:1397] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:33:11.055956 19070 raft_consensus.cc:348] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.055956 19071 raft_consensus.cc:348] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.056197 19070 raft_consensus.cc:374] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:11.056198 19071 raft_consensus.cc:374] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:11.056233 19070 raft_consensus.cc:729] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a65a6b2f6884f049a7080a796ccfba9, State: Initialized, Role: FOLLOWER
I20260620 14:33:11.056244 19071 raft_consensus.cc:729] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b4e050ec4e490eb58fe2f575432300, State: Initialized, Role: FOLLOWER
I20260620 14:33:11.056681 19071 consensus_queue.cc:260] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.056682 19070 consensus_queue.cc:260] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.057418 18925 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
I20260620 14:33:11.057631 19057 heartbeater.cc:503] Master 127.4.58.126:36951 was elected leader, sending a full tablet report...
I20260620 14:33:11.057693 19071 ts_tablet_manager.cc:1428] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:33:11.057709 19070 ts_tablet_manager.cc:1428] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260620 14:33:11.059383 18926 tablet.cc:2369] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:11.061359 19074 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:11.061515 19074 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.062537 19074 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
I20260620 14:33:11.066480 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.062744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.4.58.67:58657)
0620 14:33:11.062845 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.062859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:11.062934 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:11.063189 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:11.063191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:11.063213 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:11.063333 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.063340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.064173 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.064176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.065197 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.065206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.066023 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.066026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.066159 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.066185 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:33:11.066267 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:33:11.066339 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"spinlock_wait_cycles":1408}
I20260620 14:33:11.066514 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.062814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:58657 (local address 127.4.58.66:45839)
0620 14:33:11.062930 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.062934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.062949 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:33:11.063033 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.063036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.063086 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.063164 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:11.063443 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.064070 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.065318 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.065922 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.066013 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.066066 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.066313 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.066399 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:33:11.067299 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.062812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.4.58.67:50865)
0620 14:33:11.063119 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.063129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:11.063205 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:11.063684 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:11.063687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:11.063707 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:11.063819 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.063825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.064726 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.064728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.065837 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:11.065849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.066939 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.066942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:11.067031 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.067050 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:33:11.067125 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:33:11.067193 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260620 14:33:11.067247 18879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7b44a33bae4dbfb29799ad25a9d557" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a65a6b2f6884f049a7080a796ccfba9" is_pre_election: true
I20260620 14:33:11.067344 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.063295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:50865 (local address 127.4.58.65:35357)
0620 14:33:11.063451 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.063455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.063467 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:33:11.063514 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.063516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.063563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.063641 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:11.063920 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.064588 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.065970 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.066648 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.066743 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.066876 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.067148 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.067229 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:33:11.067557 18879 raft_consensus.cc:2454] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 0.
I20260620 14:33:11.068024 18945 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300; no voters:
I20260620 14:33:11.068274 18741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7b44a33bae4dbfb29799ad25a9d557" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db61d9fb63554ee7a875f7db0849df20" is_pre_election: true
I20260620 14:33:11.068521 19074 raft_consensus.cc:2781] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:11.068552 18741 raft_consensus.cc:2454] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 0.
I20260620 14:33:11.068615 19074 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:11.068667 19074 raft_consensus.cc:3037] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:11.069767 19074 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.070076 19074 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 election: Requested vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357)
I20260620 14:33:11.070457 18879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7b44a33bae4dbfb29799ad25a9d557" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a65a6b2f6884f049a7080a796ccfba9"
I20260620 14:33:11.070472 18741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f7b44a33bae4dbfb29799ad25a9d557" candidate_uuid: "a6b4e050ec4e490eb58fe2f575432300" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db61d9fb63554ee7a875f7db0849df20"
I20260620 14:33:11.070593 18879 raft_consensus.cc:3037] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:11.070616 18741 raft_consensus.cc:3037] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:11.071621 18741 raft_consensus.cc:2454] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 1.
I20260620 14:33:11.071743 18879 raft_consensus.cc:2454] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6b4e050ec4e490eb58fe2f575432300 in term 1.
I20260620 14:33:11.071903 18947 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6b4e050ec4e490eb58fe2f575432300, db61d9fb63554ee7a875f7db0849df20; no voters:
I20260620 14:33:11.072079 19074 raft_consensus.cc:2781] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:11.072319 19074 raft_consensus.cc:686] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 1 LEADER]: Becoming Leader. State: Replica: a6b4e050ec4e490eb58fe2f575432300, State: Running, Role: LEADER
I20260620 14:33:11.072643 19074 consensus_queue.cc:237] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.075879 18596 catalog_manager.cc:5495] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "a6b4e050ec4e490eb58fe2f575432300" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } health_report { overall_health: UNKNOWN } } }
I20260620 14:33:11.095469 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.091967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.4.58.67:38873)
0620 14:33:11.092121 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:11.092126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:11.092140 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:33:11.092226 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:11.092228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:11.092309 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:11.092407 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:11.092654 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.093200 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.094029 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:11.094244 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:11.094332 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:11.094646 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:11.095062 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:11.095172 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:11.095269 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:33:11.095330 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:33:11.101406 18879 raft_consensus.cc:1261] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Refusing update from remote peer a6b4e050ec4e490eb58fe2f575432300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:11.101403 18741 raft_consensus.cc:1261] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Refusing update from remote peer a6b4e050ec4e490eb58fe2f575432300: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:11.102131 19077 consensus_queue.cc:1035] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:11.102324 19074 consensus_queue.cc:1035] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:11.108072 19079 mvcc.cc:205] Tried to move back new op lower bound from 7298932699544084480 to 7298932699436482560. Current Snapshot: MvccSnapshot[applied={T|T < 7298932699544084480}]
I20260620 14:33:11.116782 18598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:11.116982 18598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:11.119498 18598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Sending DeleteTablet for 3 replicas of tablet 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:11.120400 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18637
I20260620 14:33:11.120335 18859 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7b44a33bae4dbfb29799ad25a9d557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:11 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:36106
I20260620 14:33:11.120599 18991 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f7b44a33bae4dbfb29799ad25a9d557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:11 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:49814
I20260620 14:33:11.121271 19091 tablet_replica.cc:330] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: stopping tablet replica
I20260620 14:33:11.121587 19091 raft_consensus.cc:2229] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:11.121925 19091 raft_consensus.cc:2258] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:11.122766 19090 tablet_replica.cc:330] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: stopping tablet replica
I20260620 14:33:11.123131 19090 raft_consensus.cc:2229] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:11.123380 19091 ts_tablet_manager.cc:1905] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:11.123569 19090 raft_consensus.cc:2258] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:11.126276 19090 ts_tablet_manager.cc:1905] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:11.126474 19091 ts_tablet_manager.cc:1918] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:11.126577 19091 log.cc:1196] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:11.126904 19091 ts_tablet_manager.cc:1939] T 4f7b44a33bae4dbfb29799ad25a9d557 P a6b4e050ec4e490eb58fe2f575432300: Deleting consensus metadata
W20260620 14:33:11.127525 18586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:35357 (error 108)
I20260620 14:33:11.128145 18586 catalog_manager.cc:4841] TS a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): tablet 4f7b44a33bae4dbfb29799ad25a9d557 (table test-table [id=73015a47a4a44b1b84d8455ba6a59de9]) successfully deleted
I20260620 14:33:11.128670 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18791
I20260620 14:33:11.130954 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.130662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:35357 (local address 127.0.0.1:58146)
0620 14:33:11.130810 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.130878 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:33:11.131168 18586 catalog_manager.cc:4584] TS db61d9fb63554ee7a875f7db0849df20 (127.4.58.65:35357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f7b44a33bae4dbfb29799ad25a9d557: Network error: Client connection negotiation failed: client connection to 127.4.58.65:35357: connect: Connection refused (error 111)
I20260620 14:33:11.135540 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18928
I20260620 14:33:11.135754 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.135455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.0.0.1:36120)
0620 14:33:11.135601 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.135666 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:45839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:33:11.140578 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 18566
I20260620 14:33:11.146229 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--webserver_interface=127.4.58.126
--webserver_port=46561
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:36951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:11.255688 19093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:11.255975 19093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:11.256031 19093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:11.260084 19093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:11.260170 19093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:11.260195 19093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:11.260219 19093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:11.260238 19093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:11.265722 19093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:36951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:11.266824 19093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:11.267863 19093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:11.277364 19098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.277364 19099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.277364 19101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.277410 19093 server_base.cc:1034] running on GCE node
I20260620 14:33:11.278339 19093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:11.279563 19093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:11.280779 19093 hybrid_clock.cc:648] HybridClock initialized: now 1781965991280736 us; error 35 us; skew 500 ppm
Jun 20 14:33:11 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:11.283666 19093 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:11.284704 19093 webserver.cc:460] Webserver started at http://127.4.58.126:46561/ using document root <none> and password file <none>
I20260620 14:33:11.285300 19093 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:11.285370 19093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:11.289426 19093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:33:11.291795 19108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.293201 19093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:11.293318 19093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e754f05539bc43148b4926445964406f"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "bd3818b9160dcecac16f1d2c4a51a42c"
server_key_iv: "e08b87b53c78d9d065a12e200c146afa"
server_key_version: "encryptionkey@0"
I20260620 14:33:11.293890 19093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:11.310947 19093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:11.314025 19093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:11.314199 19093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:11.321455 19093 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:36951
I20260620 14:33:11.321720 19160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:36951 every 8 connection(s)
I20260620 14:33:11.322690 19093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:11.327044 19161 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:11.328722 19161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap starting.
I20260620 14:33:11.332463 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19093
I20260620 14:33:11.332964 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:35357
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=46355
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:11.347303 19161 log.cc:824] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:11.358928 19161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:11.359328 19161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f: Bootstrap complete.
I20260620 14:33:11.363314 19161 raft_consensus.cc:348] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:11.363715 19161 raft_consensus.cc:729] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e754f05539bc43148b4926445964406f, State: Initialized, Role: FOLLOWER
I20260620 14:33:11.364238 19161 consensus_queue.cc:260] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:11.364362 19161 raft_consensus.cc:388] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:11.364423 19161 raft_consensus.cc:482] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:11.364518 19161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:33:11.366482 19161 raft_consensus.cc:504] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:11.366808 19161 leader_election.cc:302] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e754f05539bc43148b4926445964406f; no voters:
I20260620 14:33:11.367113 19161 leader_election.cc:288] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:33:11.367269 19165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:33:11.367493 19165 raft_consensus.cc:686] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [term 3 LEADER]: Becoming Leader. State: Replica: e754f05539bc43148b4926445964406f, State: Running, Role: LEADER
I20260620 14:33:11.367796 19165 consensus_queue.cc:237] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } }
I20260620 14:33:11.368453 19161 sys_catalog.cc:564] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:11.369199 19166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:11.369331 19166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:11.369604 19167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e754f05539bc43148b4926445964406f. Latest consensus state: current_term: 3 leader_uuid: "e754f05539bc43148b4926445964406f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e754f05539bc43148b4926445964406f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36951 } } }
I20260620 14:33:11.369690 19167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e754f05539bc43148b4926445964406f [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:11.370111 19173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:11.375306 19173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:11.376051 19173 catalog_manager.cc:1259] Loaded cluster ID: b1d9343b43ad4f8f809b5da1e976a06f
I20260620 14:33:11.376124 19173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:33:11.377192 19173 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f18017d59ad google::LogMessage::Fail() at ??:0
@ 0x7f18017d888a google::LogMessage::SendToLog() at ??:0
@ 0x7f18017d54de google::LogMessage::Flush() at ??:0
@ 0x7f18017d6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f180c114fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f180c115d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f180c112579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f180c156640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1803a273d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1803a27c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1803a293f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1803a18512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f180505c6db start_thread at ??:0
@ 0x7f18002d361f clone at ??:0
W20260620 14:33:11.460232 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:11.460474 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:11.460520 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:11.464810 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:11.464876 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:11.464965 19163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:11.470528 19163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:35357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=46355
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:11.471649 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:11.472695 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:11.482578 19202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.482640 19205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.482613 19203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.483341 19163 server_base.cc:1034] running on GCE node
I20260620 14:33:11.483848 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:11.484671 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:11.485883 19163 hybrid_clock.cc:648] HybridClock initialized: now 1781965991485848 us; error 44 us; skew 500 ppm
Jun 20 14:33:11 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:11.489434 19163 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:11.490656 19163 webserver.cc:460] Webserver started at http://127.4.58.65:46355/ using document root <none> and password file <none>
I20260620 14:33:11.491423 19163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:11.491519 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:11.497002 19163 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:33:11.500011 19212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.501786 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:11.501933 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "db61d9fb63554ee7a875f7db0849df20"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "1e71fdef2c276f3d88aecc1a4b549149"
server_key_iv: "bcbe403a094b8620cb4df80b81c63741"
server_key_version: "encryptionkey@0"
I20260620 14:33:11.502720 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:11.533488 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:11.537868 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:11.538056 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:11.538774 19163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:11.540143 19219 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:11.544366 19163 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:11.544447 19163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 14:33:11.544505 19163 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:11.546460 19163 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:11.546516 19163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:33:11.546878 19219 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Bootstrap starting.
I20260620 14:33:11.556269 19163 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:35357
I20260620 14:33:11.556579 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:35357 every 8 connection(s)
I20260620 14:33:11.557358 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:11.560336 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19163
I20260620 14:33:11.560741 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:45839
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=46839
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:11.569646 19219 log.cc:824] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:11.579371 19219 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:11.579835 19219 tablet_bootstrap.cc:492] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Bootstrap complete.
I20260620 14:33:11.581243 19219 ts_tablet_manager.cc:1397] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.007s
I20260620 14:33:11.584681 19219 raft_consensus.cc:348] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.585235 19219 raft_consensus.cc:729] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db61d9fb63554ee7a875f7db0849df20, State: Initialized, Role: FOLLOWER
I20260620 14:33:11.585772 19219 consensus_queue.cc:260] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:11.586827 19219 ts_tablet_manager.cc:1428] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 14:33:11.659343 19329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:33:11.559357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:40225)
0620 14:33:11.559858 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.559912 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:11.561908 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:11.658335 (+ 96427us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
W20260620 14:33:11.659940 19327 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:36951 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:36951: Client connection negotiation failed: client connection to 127.4.58.126:36951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:33:11.660772 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.660459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:51969)
0620 14:33:11.660599 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.660680 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:33:11.661515 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.661305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:39279)
0620 14:33:11.661411 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.661462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:33:11.661679 19327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:11.675220 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:11.675473 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:11.675515 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:11.679629 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:11.679690 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:11.679749 19330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:11.685096 19330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:45839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=46839
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:11.686110 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:11.687076 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:11.696568 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.696576 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.696714 19330 server_base.cc:1034] running on GCE node
W20260620 14:33:11.696837 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.697307 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:11.697997 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:11.699213 19330 hybrid_clock.cc:648] HybridClock initialized: now 1781965991699192 us; error 36 us; skew 500 ppm
Jun 20 14:33:11 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:11.701915 19330 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:11.702786 19330 webserver.cc:460] Webserver started at http://127.4.58.66:46839/ using document root <none> and password file <none>
I20260620 14:33:11.703330 19330 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:11.703397 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:11.707217 19330 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:33:11.709466 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.710722 19330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:33:11.710829 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "0a65a6b2f6884f049a7080a796ccfba9"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "2695a1a1bf3ebf6fbc140a0d3d50cbdd"
server_key_iv: "cecd2c7ef1123967c82a87afa6eeddeb"
server_key_version: "encryptionkey@0"
I20260620 14:33:11.711369 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:11.753870 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:11.756971 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:11.757138 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:11.757697 19330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:11.758767 19354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 14:33:11.762980 19354 ts_tablet_manager.cc:1852] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 14:33:11.763420 19354 ts_tablet_manager.cc:1905] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:11.767501 19354 ts_tablet_manager.cc:1918] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:11.767637 19354 log.cc:1196] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:11.767959 19354 ts_tablet_manager.cc:1939] T 4f7b44a33bae4dbfb29799ad25a9d557 P 0a65a6b2f6884f049a7080a796ccfba9: Deleting consensus metadata
I20260620 14:33:11.768280 19330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 14:33:11.768347 19330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260620 14:33:11.768416 19330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:11.768446 19330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.777737 19330 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:45839
I20260620 14:33:11.777829 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:45839 every 8 connection(s)
I20260620 14:33:11.778718 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:11.779640 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19330
I20260620 14:33:11.780023 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:38873
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:36951
--webserver_port=40849
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:32829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:11.782966 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.781244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:33991)
0620 14:33:11.781855 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.782004 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
W20260620 14:33:11.783404 19462 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:36951 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:36951: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
I20260620 14:33:11.784032 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.783816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:44257)
0620 14:33:11.783931 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.783980 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:33:11.784720 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.784445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:45577)
0620 14:33:11.784569 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.784653 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:33:11.784901 19462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:33:11.896013 19464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:11.896318 19464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:11.896368 19464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:11.900501 19464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:11.900578 19464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:11.900638 19464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:11.906468 19464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:32829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:38873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=40849
--tserver_master_addrs=127.4.58.126:36951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:11.907567 19464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:11.908648 19464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:11.918274 19470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.918424 19464 server_base.cc:1034] running on GCE node
W20260620 14:33:11.918274 19473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:11.918274 19471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:11.918921 19464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:11.919557 19464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:11.920732 19464 hybrid_clock.cc:648] HybridClock initialized: now 1781965991920704 us; error 39 us; skew 500 ppm
Jun 20 14:33:11 dist-test-slave-gqzk krb5kdc[18019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:11.923561 19464 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:11.924450 19464 webserver.cc:460] Webserver started at http://127.4.58.67:40849/ using document root <none> and password file <none>
I20260620 14:33:11.925040 19464 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:11.925107 19464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:11.929093 19464 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:33:11.931231 19480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.932488 19464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:11.932614 19464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "a6b4e050ec4e490eb58fe2f575432300"
format_stamp: "Formatted at 2026-06-20 14:33:09 on dist-test-slave-gqzk"
server_key: "6d8f668f19fd5560df4efd2fefee68a4"
server_key_iv: "963dc48e5a1a3445e5753b94e426679e"
server_key_version: "encryptionkey@0"
I20260620 14:33:11.933233 19464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:11.960966 19464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:11.964553 19464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:11.964740 19464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:11.965399 19464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:11.966527 19464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:11.966591 19464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.966658 19464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:11.966692 19464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:11.977284 19464 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:38873
I20260620 14:33:11.977311 19593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:38873 every 8 connection(s)
I20260620 14:33:11.978322 19464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:11.982023 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.980491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:41253)
0620 14:33:11.981177 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.981291 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":111,"threads_started":1}
W20260620 14:33:11.982466 19594 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:36951 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:36951: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
I20260620 14:33:11.983034 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.982835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:41459)
0620 14:33:11.982933 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.982980 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:33:11.983630 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:11.983445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:51591)
0620 14:33:11.983527 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:11.983564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:33:11.983803 19594 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:33:11.987964 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19464
I20260620 14:33:12.662832 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:12.662352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:59913)
0620 14:33:12.662645 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:12.662709 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260620 14:33:12.785894 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:12.785499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:38615)
0620 14:33:12.785743 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:12.785803 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260620 14:33:12.984738 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:12.984337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:37511)
0620 14:33:12.984574 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:12.984629 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260620 14:33:13.047854 19601 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:13.048161 19601 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:13.049401 19601 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
Jun 20 14:33:13 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.66@KRBTEST.COM
Jun 20 14:33:13 dist-test-slave-gqzk krb5kdc[18019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965991, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.67@KRBTEST.COM
I20260620 14:33:13.062743 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.049803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:35261 (local address 127.4.58.66:45839)
0620 14:33:13.050157 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:13.050170 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:13.050206 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:33:13.050910 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:13.050915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:13.051270 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:13.051543 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:13.052825 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.053764 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055013 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.055473 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055601 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:13.058261 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:13.058267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:13.058279 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:13.058309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:13.060248 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:13.060809 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:13.060816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:13.060821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:13.060941 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:13.061259 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:13.061263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:13.061264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:13.061669 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:13.061788 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:13.062295 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 14:33:13.062548 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":65,"threads_started":1}
I20260620 14:33:13.062736 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.049617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:45839 (local address 127.4.58.65:35261)
0620 14:33:13.049794 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:13.049809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:13.049913 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:13.051637 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:13.051643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:13.052129 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:13.052612 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:13.052626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.053913 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.053916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:13.054888 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:13.054898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.055595 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:13.055787 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:13.056422 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:13.056434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:13.058013 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:13.060372 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:13.060378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:13.060390 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:13.060669 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:13.061035 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:13.061038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:13.061040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:13.061173 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:13.061779 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:13.061785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:13.062001 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 14:33:13.062165 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:33:13.062530 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":170,"spinlock_wait_cycles":25984}
I20260620 14:33:13.062791 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.049634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:38873 (local address 127.4.58.65:59101)
0620 14:33:13.049864 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:13.049875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:13.049943 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:13.051899 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:13.051901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:13.052131 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:13.052611 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:13.052626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.053905 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.053908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:13.054890 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:13.054899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.055634 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:13.055788 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:13.056371 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:13.056386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:13.058355 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:13.060818 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:13.060821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:13.060823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:13.061020 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:13.061345 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:13.061348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:13.061350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:13.061426 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:13.062079 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:13.062083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:13.062168 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:33:13.062256 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:33:13.062613 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":276,"spinlock_wait_cycles":4352,"thread_start_us":83,"threads_started":1}
I20260620 14:33:13.062832 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.049933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:59101 (local address 127.4.58.67:38873)
0620 14:33:13.050199 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:13.050218 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:13.050252 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:33:13.050905 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:13.050911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:13.051289 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:13.051528 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:13.052808 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.053765 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055046 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:13.055505 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:13.055673 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:13.058567 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:13.058574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:13.058588 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:13.058624 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:13.060692 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:13.061135 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:13.061140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:13.061143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:13.061257 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:13.061504 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:13.061508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:13.061509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:13.061916 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:13.062031 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:13.062406 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 14:33:13.062669 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
W20260620 14:33:13.065246 19216 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
W20260620 14:33:13.065518 19214 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:13.065616 19214 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db61d9fb63554ee7a875f7db0849df20; no voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300
I20260620 14:33:13.065884 19601 raft_consensus.cc:2726] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:13.664204 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.663715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:39615)
0620 14:33:13.664018 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:13.664090 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
I20260620 14:33:13.787037 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.786617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:34107)
0620 14:33:13.786865 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:13.786925 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260620 14:33:13.985886 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:13.985466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:46279)
0620 14:33:13.985736 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:13.985794 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":70,"threads_started":1}
I20260620 14:33:14.605751 19608 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:14.605888 19608 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:14.606326 19608 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
W20260620 14:33:14.607203 19214 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
W20260620 14:33:14.607349 19216 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:14.607436 19216 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db61d9fb63554ee7a875f7db0849df20; no voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300
I20260620 14:33:14.607591 19608 raft_consensus.cc:2726] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:14.665654 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:14.665146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:44181)
0620 14:33:14.665460 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:14.665528 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260620 14:33:14.788342 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:14.787852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:59209)
0620 14:33:14.788155 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:14.788216 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":104,"threads_started":1}
I20260620 14:33:14.987143 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:14.986685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:49839)
0620 14:33:14.986962 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:14.987019 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260620 14:33:15.667075 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:15.666557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:45805)
0620 14:33:15.666861 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:15.666926 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260620 14:33:15.789659 19613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:15.789211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:42811)
0620 14:33:15.789475 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:15.789540 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260620 14:33:15.988425 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:15.987995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:36707)
0620 14:33:15.988269 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:15.988328 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
I20260620 14:33:16.668602 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:16.668100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:41139)
0620 14:33:16.668428 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:16.668492 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":72,"threads_started":1}
I20260620 14:33:16.791059 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:16.790580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:46527)
0620 14:33:16.790890 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:16.790960 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260620 14:33:16.989813 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:16.989304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:37129)
0620 14:33:16.989649 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:16.989715 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
I20260620 14:33:17.410257 19618 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:17.410400 19618 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:17.410828 19618 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
W20260620 14:33:17.411576 19214 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
W20260620 14:33:17.411733 19216 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:17.411827 19216 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db61d9fb63554ee7a875f7db0849df20; no voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300
I20260620 14:33:17.411993 19618 raft_consensus.cc:2726] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:17.669896 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:17.669428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:34875)
0620 14:33:17.669721 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:17.669787 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260620 14:33:17.792414 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:17.791930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:58331)
0620 14:33:17.792250 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:17.792316 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
I20260620 14:33:17.991219 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:17.990698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:42567)
0620 14:33:17.991044 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:17.991114 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260620 14:33:18.671190 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:18.670717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:58661)
0620 14:33:18.671005 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:18.671068 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260620 14:33:18.793764 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:18.793263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:50787)
0620 14:33:18.793599 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:18.793663 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260620 14:33:18.992535 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:18.992054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:54973)
0620 14:33:18.992380 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:18.992440 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260620 14:33:19.074998 19625 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:19.075146 19625 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:19.075578 19625 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
W20260620 14:33:19.076401 19214 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
W20260620 14:33:19.076532 19216 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:19.076640 19216 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db61d9fb63554ee7a875f7db0849df20; no voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300
I20260620 14:33:19.076808 19625 raft_consensus.cc:2726] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:19.672323 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:19.671909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:39333)
0620 14:33:19.672168 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:19.672224 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260620 14:33:19.794946 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:19.794513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:39953)
0620 14:33:19.794782 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:19.794849 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260620 14:33:19.993775 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:19.993329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:52749)
0620 14:33:19.993611 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:19.993676 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260620 14:33:20.673604 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:20.673137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:40061)
0620 14:33:20.673433 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:20.673490 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260620 14:33:20.796094 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:20.795667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:48525)
0620 14:33:20.795930 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:20.795990 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260620 14:33:20.834231 19631 raft_consensus.cc:482] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:20.834369 19631 raft_consensus.cc:504] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a65a6b2f6884f049a7080a796ccfba9" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 45839 } } peers { permanent_uuid: "a6b4e050ec4e490eb58fe2f575432300" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 38873 } } peers { permanent_uuid: "db61d9fb63554ee7a875f7db0849df20" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 35357 } }
I20260620 14:33:20.834811 19631 leader_election.cc:288] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839), a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873)
W20260620 14:33:20.835582 19216 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a6b4e050ec4e490eb58fe2f575432300 (127.4.58.67:38873): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
W20260620 14:33:20.835705 19214 leader_election.cc:341] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0a65a6b2f6884f049a7080a796ccfba9 (127.4.58.66:45839): Not found: Tablet not found: 4f7b44a33bae4dbfb29799ad25a9d557
I20260620 14:33:20.835793 19214 leader_election.cc:302] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db61d9fb63554ee7a875f7db0849df20; no voters: 0a65a6b2f6884f049a7080a796ccfba9, a6b4e050ec4e490eb58fe2f575432300
I20260620 14:33:20.835935 19631 raft_consensus.cc:2726] T 4f7b44a33bae4dbfb29799ad25a9d557 P db61d9fb63554ee7a875f7db0849df20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:33:20.994988 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:20.994549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:47345)
0620 14:33:20.994828 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:20.994889 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260620 14:33:21.674897 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:21.674457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:52101)
0620 14:33:21.674735 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:21.674798 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
I20260620 14:33:21.797290 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:21.796905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:39903)
0620 14:33:21.797137 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:21.797193 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260620 14:33:21.996380 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:21.995866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:39949)
0620 14:33:21.996215 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:21.996276 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":145,"threads_started":1}
I20260620 14:33:22.676190 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:22.675702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:49093)
0620 14:33:22.676015 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:22.676081 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":78,"threads_started":1}
I20260620 14:33:22.798449 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:22.798041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:35909)
0620 14:33:22.798302 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:22.798355 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260620 14:33:22.997629 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:22.997189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:49855)
0620 14:33:22.997468 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:22.997530 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260620 14:33:23.677521 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:23.677022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:34409)
0620 14:33:23.677348 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:23.677417 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":92,"threads_started":1}
I20260620 14:33:23.799640 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:23.799200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:60141)
0620 14:33:23.799450 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:23.799523 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260620 14:33:23.998858 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:23.998415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:37601)
0620 14:33:23.998701 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:23.998760 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260620 14:33:24.678907 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:24.678380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:35575)
0620 14:33:24.678725 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:24.678794 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260620 14:33:24.800984 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:24.800476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:56875)
0620 14:33:24.800816 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:24.800884 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260620 14:33:25.000056 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:24.999630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:41855)
0620 14:33:24.999888 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:24.999948 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260620 14:33:25.680346 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:25.679833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:51017)
0620 14:33:25.680123 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:25.680238 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260620 14:33:25.802316 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:25.801862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:48107)
0620 14:33:25.802158 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:25.802219 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260620 14:33:26.001332 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:26.000861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.67:40909)
0620 14:33:26.001163 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:26.001228 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260620 14:33:26.681942 19648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:26.681409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.65:44307)
0620 14:33:26.681757 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:26.681836 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
I20260620 14:33:26.803794 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:26.803295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36951 (local address 127.4.58.66:40235)
0620 14:33:26.803569 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:26.803648 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:36951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260620 14:33:26.991091 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19163
I20260620 14:33:26.996096 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19330
I20260620 14:33:27.000943 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19464
2026-06-20T14:33:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19005 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:33:27 dist-test-slave-gqzk krb5kdc[19652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:33:27 dist-test-slave-gqzk krb5kdc[19652](info): set up 2 sockets
Jun 20 14:33:27 dist-test-slave-gqzk krb5kdc[19652](info): commencing operation
krb5kdc: starting...
W20260620 14:33:27.956730 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:33:28.075856 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:28.187338 19665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:28.187655 19665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:28.187728 19665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:28.191825 19665 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:33:28.191895 19665 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:33:28.191917 19665 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:33:28.191936 19665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:28.191957 19665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:28.191982 19665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:28.192008 19665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:28.192039 19665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:28.197640 19665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:28.198818 19665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:28.199934 19665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:28.208484 19673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.208458 19671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.208534 19670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:28.208617 19665 server_base.cc:1034] running on GCE node
I20260620 14:33:28.209316 19665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:28.210436 19665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:28.211597 19665 hybrid_clock.cc:648] HybridClock initialized: now 1781966008211580 us; error 32 us; skew 500 ppm
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:28.214303 19665 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:28.215260 19665 webserver.cc:460] Webserver started at http://127.4.58.126:46771/ using document root <none> and password file <none>
I20260620 14:33:28.215811 19665 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:28.215878 19665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:28.216069 19665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:28.218180 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "aa91e24392844917ae3845b31c4ab028"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "3a50fcd6515336a3569c418ef408f075"
server_key_iv: "06016e27b33e05d24c4b6cb95c387bfb"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.218742 19665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "aa91e24392844917ae3845b31c4ab028"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "3a50fcd6515336a3569c418ef408f075"
server_key_iv: "06016e27b33e05d24c4b6cb95c387bfb"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.222498 19665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:33:28.224970 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.226269 19665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:33:28.226387 19665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "aa91e24392844917ae3845b31c4ab028"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "3a50fcd6515336a3569c418ef408f075"
server_key_iv: "06016e27b33e05d24c4b6cb95c387bfb"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.226490 19665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:28.238560 19665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:28.241621 19665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:28.241822 19665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:28.248871 19665 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:39907
I20260620 14:33:28.249140 19732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:39907 every 8 connection(s)
I20260620 14:33:28.249846 19665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:28.252005 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19665
I20260620 14:33:28.252099 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:33:28.252377 4329 external_mini_cluster.cc:1428] Setting key 107ad6fc7b791c897cb66ba4de22da5f
I20260620 14:33:28.252928 19733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:28.260031 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap starting.
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:28.263070 19733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:28.263929 19733 log.cc:824] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:28.266371 19733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: No bootstrap required, opened a new log
I20260620 14:33:28.267771 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.253843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.4.58.126:39907)
0620 14:33:28.254235 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.254249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.254284 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:33:28.254952 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.254969 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.255301 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.255578 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:28.256551 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.257606 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.258877 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.259352 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.259474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.262380 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:28.262387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:28.262402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:28.262434 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:28.264293 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.264826 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.264833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.264839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.264979 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.265371 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.265374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.265375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.265898 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:28.266087 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.266432 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.266784 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260620 14:33:28.270139 19733 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:28.270414 19733 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:28.270512 19733 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Initialized, Role: FOLLOWER
I20260620 14:33:28.270972 19733 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:28.271103 19733 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:28.271150 19733 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:28.271234 19733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:28.272125 19733 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:28.272450 19733 leader_election.cc:302] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa91e24392844917ae3845b31c4ab028; no voters:
I20260620 14:33:28.272740 19733 leader_election.cc:288] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:33:28.272917 19738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:28.273181 19738 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 LEADER]: Becoming Leader. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Running, Role: LEADER
I20260620 14:33:28.273499 19738 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:28.274013 19733 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:28.275169 19740 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa91e24392844917ae3845b31c4ab028. Latest consensus state: current_term: 1 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:33:28.275311 19740 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:28.275743 19739 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:33:28.276507 19739 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
W20260620 14:33:28.278780 19753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:33:28.278846 19753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:33:28.278956 19747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:28.279874 19747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:28.285665 19747 catalog_manager.cc:1347] Generated new cluster ID: 447298d9ab84451fa928e79531796031
I20260620 14:33:28.285748 19747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:28.297871 19747 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:33:28.298870 19747 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:28.303426 19747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Generated new TSK 0
I20260620 14:33:28.304124 19747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:33:28.339890 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:28.453226 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:28.453493 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:28.453541 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:28.457589 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:28.457651 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:28.457746 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:28.463131 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:28.464236 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:28.465334 19759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:28.474958 19759 server_base.cc:1034] running on GCE node
W20260620 14:33:28.474920 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.474946 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.474996 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:28.475559 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:28.476198 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:28.477349 19759 hybrid_clock.cc:648] HybridClock initialized: now 1781966008477321 us; error 44 us; skew 500 ppm
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:28.480316 19759 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:28.481216 19759 webserver.cc:460] Webserver started at http://127.4.58.65:41593/ using document root <none> and password file <none>
I20260620 14:33:28.481747 19759 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:28.481804 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:28.481961 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:28.483932 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "168390242d8fe3c68f662ee936cbf3cc"
server_key_iv: "44a9557a3f1c17897c3a5348ea7299fe"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.484434 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "168390242d8fe3c68f662ee936cbf3cc"
server_key_iv: "44a9557a3f1c17897c3a5348ea7299fe"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.488085 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:33:28.490562 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.491842 19759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:28.491961 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "168390242d8fe3c68f662ee936cbf3cc"
server_key_iv: "44a9557a3f1c17897c3a5348ea7299fe"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.492090 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:28.503429 19759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:28.506361 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:28.506529 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:28.507097 19759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:28.507895 19759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:28.507941 19759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.507980 19759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:28.508009 19759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.516367 19759 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:44939
I20260620 14:33:28.516412 19887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:44939 every 8 connection(s)
I20260620 14:33:28.517306 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:28.526361 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19759
I20260620 14:33:28.526479 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:33:28.526751 4329 external_mini_cluster.cc:1428] Setting key 3ca9ba0e07a5c9eca54c04c31ce1d9e6
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:28.531603 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.519184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:35175 (local address 127.4.58.126:39907)
0620 14:33:28.519321 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.519325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.520734 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 14:33:28.521702 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.521705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.521762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.521837 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:28.523008 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.523543 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.524683 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.524981 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.525076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.527557 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:28.527560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:28.527562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:28.527587 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:28.529164 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.529760 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.529764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.529766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.529865 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.530289 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.530292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.530293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.530522 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:28.530622 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.531293 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.531434 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:33:28.532523 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.519608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:35175)
0620 14:33:28.520554 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.520621 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.521475 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.522029 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.522036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.522390 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:28.522868 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.522881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.523657 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.523660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.524570 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.524579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.525133 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.525138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.525302 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.525877 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:28.525894 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:28.527399 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:28.529316 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.529325 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.529344 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.529672 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.529961 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.529964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.529967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.530200 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.530651 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:28.530658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:28.531009 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 14:33:28.531217 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 14:33:28.531574 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":554,"threads_started":1}
I20260620 14:33:28.533958 19888 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:28.534248 19888 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:28.534863 19888 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:33:28.536652 19697 ts_manager.cc:194] Registered new tserver with Master: 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:28.537909 19697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:35175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:33:28.555636 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:28.665500 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:28.665789 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:28.665853 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:28.669938 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:28.670017 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:28.670100 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:28.675772 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:28.676995 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:28.678143 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:33:28.688041 19893 server_base.cc:1034] running on GCE node
W20260620 14:33:28.687963 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.687945 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.687990 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:28.688742 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:28.689513 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:28.690703 19893 hybrid_clock.cc:648] HybridClock initialized: now 1781966008690681 us; error 34 us; skew 500 ppm
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:28.693480 19893 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:28.694410 19893 webserver.cc:460] Webserver started at http://127.4.58.66:34681/ using document root <none> and password file <none>
I20260620 14:33:28.694972 19893 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:28.695040 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:28.695288 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:28.697468 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "04b909af50314988a3286aebeaa10139"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "e3bf745fec5a6c4a866f9b5bb538e649"
server_key_iv: "00546f98832fcf17f390d6563ffea153"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.698057 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "04b909af50314988a3286aebeaa10139"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "e3bf745fec5a6c4a866f9b5bb538e649"
server_key_iv: "00546f98832fcf17f390d6563ffea153"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.701984 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:33:28.704478 19908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.705852 19893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:33:28.705986 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "04b909af50314988a3286aebeaa10139"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "e3bf745fec5a6c4a866f9b5bb538e649"
server_key_iv: "00546f98832fcf17f390d6563ffea153"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.706135 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:28.726284 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:28.729492 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:28.729699 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:28.730352 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:28.731204 19893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:28.731253 19893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.731307 19893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:28.731331 19893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.739902 19893 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46079
I20260620 14:33:28.739931 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46079 every 8 connection(s)
I20260620 14:33:28.740885 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:28.742163 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 19893
I20260620 14:33:28.742287 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:33:28.742544 4329 external_mini_cluster.cc:1428] Setting key c9955e75c6704660ac45b1719f12cc63
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:33:28.754427 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.742732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:42531 (local address 127.4.58.126:39907)
0620 14:33:28.742888 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.742892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.743978 (+ 1086us) server_negotiation.cc:407] Connection header received
0620 14:33:28.744994 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.744997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.745050 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.745137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:28.746370 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.747081 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.748274 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.748607 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.748730 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.751092 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:28.751096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:28.751099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:28.751127 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:28.752543 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.753063 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.753066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.753068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.753152 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.753470 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.753472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.753473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.753655 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:28.753733 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.754193 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.754294 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:33:28.755429 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.743131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:42531)
0620 14:33:28.743836 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.743895 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.744733 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.745339 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.745346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.745730 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:28.746229 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.746243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.747222 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.747225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.748154 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.748163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.748805 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.748810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.748956 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.749500 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:28.749516 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:28.750916 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:28.752656 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.752661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.752681 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.752976 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.753237 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.753239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.753240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.753385 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.753741 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:28.753747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:28.753997 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:33:28.754148 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:33:28.754455 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":140,"threads_started":1}
I20260620 14:33:28.756705 20022 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:28.757004 20022 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:28.757567 20022 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
I20260620 14:33:28.758698 19697 ts_manager.cc:194] Registered new tserver with Master: 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079)
I20260620 14:33:28.759464 19697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:42531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:33:28.769613 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:28.882154 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:28.882438 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:28.882483 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:28.886781 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:28.886847 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:28.886909 20027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:28.892329 20027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:28.893507 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:28.894544 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:28.904609 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:28.904603 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:28.904690 20027 server_base.cc:1034] running on GCE node
W20260620 14:33:28.904574 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:28.905418 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:28.906121 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:28.907296 20027 hybrid_clock.cc:648] HybridClock initialized: now 1781966008907263 us; error 47 us; skew 500 ppm
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:28.910478 20027 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:28.911412 20027 webserver.cc:460] Webserver started at http://127.4.58.67:46187/ using document root <none> and password file <none>
I20260620 14:33:28.911978 20027 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:28.912045 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:28.912232 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:33:28.914338 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "5a48e778682740cabd253bd8c211cbb6"
server_key_iv: "13cdd5f8c78fb2fc78ad6373abc7d9de"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.914881 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "5a48e778682740cabd253bd8c211cbb6"
server_key_iv: "13cdd5f8c78fb2fc78ad6373abc7d9de"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.918619 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 14:33:28.921132 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.922340 20027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:33:28.922456 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "5a48e778682740cabd253bd8c211cbb6"
server_key_iv: "13cdd5f8c78fb2fc78ad6373abc7d9de"
server_key_version: "encryptionkey@0"
I20260620 14:33:28.922552 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:28.933481 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:28.936496 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:28.936671 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:28.937256 20027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:28.938086 20027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:28.938130 20027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.938180 20027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:28.938208 20027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:28.946382 20027 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40645
I20260620 14:33:28.946447 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40645 every 8 connection(s)
I20260620 14:33:28.947278 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:28.956094 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20027
Jun 20 14:33:28 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:28.956214 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:33:28.956458 4329 external_mini_cluster.cc:1428] Setting key 7062cd52420d6ae0970f11f2e83be19c
I20260620 14:33:28.960171 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.949140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:50987 (local address 127.4.58.126:39907)
0620 14:33:28.949304 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.949308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.950316 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 14:33:28.951191 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.951194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.951243 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.951313 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:28.952440 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.953022 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.954213 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.954417 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.954484 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.956818 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:28.956821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:28.956823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:28.956847 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:28.958303 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.958815 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.958817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.958819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.958891 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.959208 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.959210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.959211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.959373 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:28.959440 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.959900 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.960045 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:33:28.961059 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.949502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:50987)
0620 14:33:28.950174 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.950216 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.950992 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.951466 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.951472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.951848 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:28.952327 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.952339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.953152 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.953155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.954093 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.954102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.954551 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.954554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.954695 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.955207 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:28.955223 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:28.956628 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:28.958417 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.958423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.958438 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.958736 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.958955 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:28.958957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:28.958959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:28.959138 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:28.959459 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:28.959465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:28.959709 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 14:33:28.959862 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:33:28.960176 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":109,"threads_started":1}
I20260620 14:33:28.962093 20156 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:28.962299 20156 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:28.962759 20156 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
I20260620 14:33:28.963732 19697 ts_manager.cc:194] Registered new tserver with Master: 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
I20260620 14:33:28.964270 19697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:50987
I20260620 14:33:28.970326 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:28.979357 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.972234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.4.58.126:39907)
0620 14:33:28.972356 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.972361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.972557 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:33:28.972673 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.972677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.972721 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.972832 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:28.973522 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.974084 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.974925 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.975144 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.975237 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.976535 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:28.976540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:28.976543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:28.976567 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:28.977789 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.978197 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.978200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.978201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.978278 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:28.978519 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:28.978522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:28.978524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:28.978747 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:28.978861 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.979083 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.979205 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:33:28.982151 19697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:28.985071 19697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:29.000057 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.993856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46079 (local address 127.0.0.1:43120)
0620 14:33:28.995041 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.995054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.995131 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.995663 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.995667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.995701 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:28.995917 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.995930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997441 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.997444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.998546 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.998556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.999479 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.999596 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.999639 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:33:28.999756 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:33:28.999885 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":91,"spinlock_wait_cycles":6656,"thread_start_us":80,"threads_started":1}
I20260620 14:33:29.000051 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.994358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.4.58.66:46079)
0620 14:33:28.994659 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.994670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.995221 (+ 551us) server_negotiation.cc:407] Connection header received
0620 14:33:28.995305 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.995311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.995510 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.995672 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:28.996087 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997299 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.998672 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.999359 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999463 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.999613 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:28.999796 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:33:28.999898 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260620 14:33:29.000698 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.993809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40645 (local address 127.0.0.1:53636)
0620 14:33:28.994680 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.994730 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.994849 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.995657 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.995662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.995698 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:28.995852 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.995865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997519 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.997522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.998898 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.998909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.000073 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.000077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.000400 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.000426 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:33:29.000489 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:33:29.000562 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":43,"spinlock_wait_cycles":4352,"thread_start_us":95,"threads_started":1}
I20260620 14:33:29.000730 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.995136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.0.0.1:38904)
0620 14:33:28.995920 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:28.995932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:28.996012 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:28.996390 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:28.996393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:28.996416 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:28.996500 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.996506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997804 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.997807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:28.998934 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:28.998942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.999971 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.000437 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.000459 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:33:29.000534 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:33:29.000605 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260620 14:33:29.001072 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.995240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.4.58.65:44939)
0620 14:33:28.995486 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.995500 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.996002 (+ 502us) server_negotiation.cc:407] Connection header received
0620 14:33:28.996109 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.996114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.996278 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.996407 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:28.996595 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997676 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999049 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.999689 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999777 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:28.999904 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:29.000860 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 14:33:29.000959 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260620 14:33:29.001472 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:28.993914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.4.58.67:40645)
0620 14:33:28.995169 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:28.995182 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:28.995206 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:33:28.995286 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:28.995291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:28.995508 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:28.995661 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:28.996008 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.997391 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999029 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:28.999781 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:28.999879 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.000011 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:29.001283 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0620 14:33:29.001364 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":186,"threads_started":1}
I20260620 14:33:29.003191 19956 tablet_service.cc:1467] Processing CreateTablet for tablet db12d8bd18154f11afd632175debabb0 (DEFAULT_TABLE table=test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:29.003731 20090 tablet_service.cc:1467] Processing CreateTablet for tablet db12d8bd18154f11afd632175debabb0 (DEFAULT_TABLE table=test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:29.004320 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db12d8bd18154f11afd632175debabb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:29.004702 20090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db12d8bd18154f11afd632175debabb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:29.006465 19822 tablet_service.cc:1467] Processing CreateTablet for tablet db12d8bd18154f11afd632175debabb0 (DEFAULT_TABLE table=test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:29.007622 19822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db12d8bd18154f11afd632175debabb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:29.011126 20173 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Bootstrap starting.
I20260620 14:33:29.013724 20173 tablet_bootstrap.cc:654] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:29.014439 20175 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap starting.
I20260620 14:33:29.014536 20173 log.cc:824] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:29.015225 20174 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Bootstrap starting.
I20260620 14:33:29.016535 20175 tablet_bootstrap.cc:654] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:29.016732 20173 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: No bootstrap required, opened a new log
I20260620 14:33:29.016973 20173 ts_tablet_manager.cc:1397] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 14:33:29.017593 20175 log.cc:824] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:29.017827 20174 tablet_bootstrap.cc:654] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:29.018648 20174 log.cc:824] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:29.019467 20175 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: No bootstrap required, opened a new log
I20260620 14:33:29.019654 20175 ts_tablet_manager.cc:1397] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:33:29.020859 20174 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: No bootstrap required, opened a new log
I20260620 14:33:29.021059 20174 ts_tablet_manager.cc:1397] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:33:29.020865 20173 raft_consensus.cc:348] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.021214 20173 raft_consensus.cc:374] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:29.021291 20173 raft_consensus.cc:729] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a41710e3d0e4daeb4e7d58ca12fa1b8, State: Initialized, Role: FOLLOWER
I20260620 14:33:29.021787 20173 consensus_queue.cc:260] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.022738 20156 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
I20260620 14:33:29.022861 20173 ts_tablet_manager.cc:1428] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 14:33:29.024567 20174 raft_consensus.cc:348] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.024827 20174 raft_consensus.cc:374] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:29.024910 20174 raft_consensus.cc:729] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b909af50314988a3286aebeaa10139, State: Initialized, Role: FOLLOWER
I20260620 14:33:29.025338 20174 consensus_queue.cc:260] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.025310 20175 raft_consensus.cc:348] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.025597 20175 raft_consensus.cc:374] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:29.025750 20175 raft_consensus.cc:729] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2bb8b360044cb1b9afc6f72693e21c, State: Initialized, Role: FOLLOWER
I20260620 14:33:29.025988 20022 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
I20260620 14:33:29.026288 20174 ts_tablet_manager.cc:1428] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:33:29.027431 20175 consensus_queue.cc:260] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.028167 19888 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
I20260620 14:33:29.028481 20175 ts_tablet_manager.cc:1428] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:33:29.035466 20179 raft_consensus.cc:482] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:29.035692 20179 raft_consensus.cc:504] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.036834 20179 leader_election.cc:288] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:29.040565 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.037131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:40545 (local address 127.4.58.65:44939)
0620 14:33:29.037234 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:29.037238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:29.037251 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:33:29.037343 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:29.037346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:29.037396 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:29.037479 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:29.037783 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.038413 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.039523 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.039992 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.040067 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.040121 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:29.040373 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:33:29.040444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:33:29.040568 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.037009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.4.58.67:40545)
0620 14:33:29.037109 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:29.037124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:29.037240 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:29.037509 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:29.037512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:29.037540 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:29.037643 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:29.037649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.038528 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.038532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.039392 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:29.039401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.040125 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.040129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.040238 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.040264 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:33:29.040352 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:33:29.040424 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:33:29.040797 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.037236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:50629 (local address 127.4.58.66:46079)
0620 14:33:29.037366 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:29.037370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:29.037694 (+ 324us) server_negotiation.cc:407] Connection header received
0620 14:33:29.037803 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:29.037806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:29.037846 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:29.037903 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:29.038151 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.038716 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.039766 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.040280 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.040372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.040429 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:29.040595 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:33:29.040665 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:33:29.040819 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.037009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46079 (local address 127.4.58.67:50629)
0620 14:33:29.037613 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:29.037624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:29.037722 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:29.037921 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:29.037924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:29.037941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:29.038056 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:29.038062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.038813 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.038815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.039629 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:29.039638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.040387 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.040390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:29.040480 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.040502 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:33:29.040572 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:33:29.040632 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":309,"thread_start_us":101,"threads_started":1}
I20260620 14:33:29.041283 19842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db12d8bd18154f11afd632175debabb0" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" is_pre_election: true
I20260620 14:33:29.041376 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db12d8bd18154f11afd632175debabb0" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b909af50314988a3286aebeaa10139" is_pre_election: true
I20260620 14:33:29.041626 19842 raft_consensus.cc:2454] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 0.
I20260620 14:33:29.041626 19976 raft_consensus.cc:2454] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 0.
I20260620 14:33:29.042130 20043 leader_election.cc:302] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a41710e3d0e4daeb4e7d58ca12fa1b8, 2e2bb8b360044cb1b9afc6f72693e21c; no voters:
I20260620 14:33:29.042400 20179 raft_consensus.cc:2781] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:29.042479 20179 raft_consensus.cc:482] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:29.042554 20179 raft_consensus.cc:3037] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:29.043547 20179 raft_consensus.cc:504] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.043881 20179 leader_election.cc:288] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 election: Requested vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:29.044215 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db12d8bd18154f11afd632175debabb0" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b909af50314988a3286aebeaa10139"
I20260620 14:33:29.044235 19842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db12d8bd18154f11afd632175debabb0" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
I20260620 14:33:29.044337 19976 raft_consensus.cc:3037] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:29.044350 19842 raft_consensus.cc:3037] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:29.045363 19842 raft_consensus.cc:2454] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 1.
I20260620 14:33:29.045369 19976 raft_consensus.cc:2454] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 1.
I20260620 14:33:29.045662 20046 leader_election.cc:302] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8; no voters:
I20260620 14:33:29.045867 20179 raft_consensus.cc:2781] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:29.046077 20179 raft_consensus.cc:686] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a41710e3d0e4daeb4e7d58ca12fa1b8, State: Running, Role: LEADER
I20260620 14:33:29.046420 20179 consensus_queue.cc:237] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:29.049541 19697 catalog_manager.cc:5495] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } health_report { overall_health: UNKNOWN } } }
I20260620 14:33:29.062769 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.059518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.4.58.67:40645)
0620 14:33:29.059623 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:29.059626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:29.059638 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:33:29.059778 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:29.059780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:29.059836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:29.059920 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:29.060109 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.060606 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.061492 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:29.061708 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:29.061780 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:29.062000 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:29.062405 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:29.062489 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:29.062535 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:33:29.062586 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:33:29.069063 19976 raft_consensus.cc:1261] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Refusing update from remote peer 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:29.069064 19842 raft_consensus.cc:1261] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Refusing update from remote peer 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:29.069723 20179 consensus_queue.cc:1035] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:29.069957 20183 consensus_queue.cc:1035] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:29.077874 20188 mvcc.cc:205] Tried to move back new op lower bound from 7298932773138644992 to 7298932773056856064. Current Snapshot: MvccSnapshot[applied={T|T < 7298932773138644992}]
I20260620 14:33:29.086592 19697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:29.086781 19697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:29.089993 19697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Sending DeleteTablet for 3 replicas of tablet db12d8bd18154f11afd632175debabb0
I20260620 14:33:29.090857 19822 tablet_service.cc:1514] Processing DeleteTablet for tablet db12d8bd18154f11afd632175debabb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:38904
I20260620 14:33:29.091463 20195 tablet_replica.cc:330] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: stopping tablet replica
I20260620 14:33:29.091921 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19759
I20260620 14:33:29.091938 20195 raft_consensus.cc:2229] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:29.092028 20090 tablet_service.cc:1514] Processing DeleteTablet for tablet db12d8bd18154f11afd632175debabb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:53636
I20260620 14:33:29.092286 20195 raft_consensus.cc:2258] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:29.092789 20196 tablet_replica.cc:330] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: stopping tablet replica
I20260620 14:33:29.092935 19956 tablet_service.cc:1514] Processing DeleteTablet for tablet db12d8bd18154f11afd632175debabb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:43120
I20260620 14:33:29.093082 20196 raft_consensus.cc:2229] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:29.093384 20197 tablet_replica.cc:330] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: stopping tablet replica
I20260620 14:33:29.093451 20196 raft_consensus.cc:2258] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:29.093765 20197 raft_consensus.cc:2229] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:29.094076 20197 raft_consensus.cc:2258] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:29.095134 20196 ts_tablet_manager.cc:1905] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:29.095520 20197 ts_tablet_manager.cc:1905] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:29.098058 20196 ts_tablet_manager.cc:1918] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:29.098186 20197 ts_tablet_manager.cc:1918] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:29.098273 20196 log.cc:1196] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/db12d8bd18154f11afd632175debabb0
I20260620 14:33:29.098361 20197 log.cc:1196] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/db12d8bd18154f11afd632175debabb0
I20260620 14:33:29.098677 20196 ts_tablet_manager.cc:1939] T db12d8bd18154f11afd632175debabb0 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting consensus metadata
I20260620 14:33:29.098759 20197 ts_tablet_manager.cc:1939] T db12d8bd18154f11afd632175debabb0 P 04b909af50314988a3286aebeaa10139: Deleting consensus metadata
I20260620 14:33:29.100040 19684 catalog_manager.cc:4841] TS 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): tablet db12d8bd18154f11afd632175debabb0 (table test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]) successfully deleted
I20260620 14:33:29.100297 19682 catalog_manager.cc:4841] TS 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): tablet db12d8bd18154f11afd632175debabb0 (table test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]) successfully deleted
W20260620 14:33:29.100456 19681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:44939 (error 108)
I20260620 14:33:29.101229 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19893
I20260620 14:33:29.101541 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:29.101236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.0.0.1:38912)
0620 14:33:29.101378 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:29.101448 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:33:29.102073 19681 catalog_manager.cc:4584] TS 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db12d8bd18154f11afd632175debabb0: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44939: connect: Connection refused (error 111)
I20260620 14:33:29.106259 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20027
I20260620 14:33:29.111545 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 19665
I20260620 14:33:49.117352 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--webserver_interface=127.4.58.126
--webserver_port=46771
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:33:49.230312 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:49.230607 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:49.230665 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:49.234740 20199 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:33:49.234803 20199 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:33:49.234824 20199 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:33:49.234841 20199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:33:49.234863 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:49.234889 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:49.234918 20199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:33:49.234941 20199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:33:49.240474 20199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:49.241721 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:49.242803 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:49.256150 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:49.256295 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.256362 20199 server_base.cc:1034] running on GCE node
W20260620 14:33:49.256150 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.257207 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:49.258471 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:49.259645 20199 hybrid_clock.cc:648] HybridClock initialized: now 1781966029259608 us; error 49 us; skew 500 ppm
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:49.266160 20199 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:33:49.267217 20199 webserver.cc:460] Webserver started at http://127.4.58.126:46771/ using document root <none> and password file <none>
I20260620 14:33:49.267827 20199 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:49.267908 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:49.272339 20199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:49.274718 20215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.276093 20199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 14:33:49.276218 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "aa91e24392844917ae3845b31c4ab028"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "3a50fcd6515336a3569c418ef408f075"
server_key_iv: "06016e27b33e05d24c4b6cb95c387bfb"
server_key_version: "encryptionkey@0"
I20260620 14:33:49.276932 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:49.300235 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:49.303824 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:49.304024 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:49.311293 20199 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:39907
I20260620 14:33:49.311576 20267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:39907 every 8 connection(s)
I20260620 14:33:49.312589 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:33:49.314189 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20199
I20260620 14:33:49.314640 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:44939
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=41593
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:49.317473 20268 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:33:49.319327 20268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap starting.
I20260620 14:33:49.336804 20268 log.cc:824] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:49.346143 20268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:49.346633 20268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap complete.
I20260620 14:33:49.352345 20268 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:49.352973 20268 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Initialized, Role: FOLLOWER
I20260620 14:33:49.353590 20268 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:49.353786 20268 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:33:49.353866 20268 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:33:49.353969 20268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:33:49.356662 20268 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:49.357081 20268 leader_election.cc:302] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa91e24392844917ae3845b31c4ab028; no voters:
I20260620 14:33:49.357463 20268 leader_election.cc:288] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:33:49.357683 20272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:33:49.358047 20272 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 LEADER]: Becoming Leader. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Running, Role: LEADER
I20260620 14:33:49.358489 20272 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:33:49.358860 20268 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:33:49.360082 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:33:49.360221 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:49.360944 20274 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa91e24392844917ae3845b31c4ab028. Latest consensus state: current_term: 2 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:33:49.361027 20274 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
I20260620 14:33:49.363690 20289 catalog_manager.cc:1259] Loaded cluster ID: 447298d9ab84451fa928e79531796031
I20260620 14:33:49.363749 20289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: loading cluster ID for follower catalog manager: success
I20260620 14:33:49.366058 20289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: acquiring CA information for follower catalog manager: success
I20260620 14:33:49.367741 20289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:33:49.367892 20281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:33:49.370533 20281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:33:49.371052 20281 catalog_manager.cc:1259] Loaded cluster ID: 447298d9ab84451fa928e79531796031
I20260620 14:33:49.371121 20281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:33:49.371898 20281 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:33:49.372431 20281 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Loaded TSK: 0
I20260620 14:33:49.395962 20281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Generated new TSK 1
I20260620 14:33:49.396283 20281 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:33:49.450517 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:49.450809 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:49.450874 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:49.455271 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:49.455343 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:49.455426 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:33:49.460987 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:44939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=41593
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:49.462162 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:49.463191 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:49.473059 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:49.473181 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:49.473129 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.473598 20270 server_base.cc:1034] running on GCE node
I20260620 14:33:49.474078 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:49.474846 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:49.476052 20270 hybrid_clock.cc:648] HybridClock initialized: now 1781966029476016 us; error 44 us; skew 500 ppm
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:49.479382 20270 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:33:49.480533 20270 webserver.cc:460] Webserver started at http://127.4.58.65:41593/ using document root <none> and password file <none>
I20260620 14:33:49.481280 20270 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:49.481389 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:49.486548 20270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:33:49.489348 20306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.490953 20270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:49.491106 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "168390242d8fe3c68f662ee936cbf3cc"
server_key_iv: "44a9557a3f1c17897c3a5348ea7299fe"
server_key_version: "encryptionkey@0"
I20260620 14:33:49.491838 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:49.511439 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:49.515321 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:49.515507 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:49.516216 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:49.517521 20313 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:33:49.521862 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:33:49.521948 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 14:33:49.522017 20270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:33:49.524349 20270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:33:49.524412 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:33:49.524806 20313 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap starting.
I20260620 14:33:49.534902 20270 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:44939
I20260620 14:33:49.535405 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:44939 every 8 connection(s)
I20260620 14:33:49.536365 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:33:49.541750 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20270
I20260620 14:33:49.542253 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:46079
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=34681
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:33:49.545372 20313 log.cc:824] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:49.554601 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.539254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:60771 (local address 127.4.58.126:39907)
0620 14:33:49.539808 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.539833 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.540449 (+ 616us) server_negotiation.cc:407] Connection header received
0620 14:33:49.542044 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.542061 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.542510 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.542892 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.544442 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.545518 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.546893 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.547370 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.547528 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.549879 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.549884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.549897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.549927 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.551463 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.551976 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.551981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.551985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.552105 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.552414 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.552416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.552418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.552812 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.552930 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.553475 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.553753 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":130,"threads_started":1}
I20260620 14:33:49.554613 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.539094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:60771)
0620 14:33:49.540215 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.540285 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.541497 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.543044 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.543054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.543579 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:49.544243 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.544263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.545670 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.545673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.546771 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.546782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.547509 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.547513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.547700 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.548278 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:49.548295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:49.549692 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:49.551599 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.551605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.551618 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.551868 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.552193 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.552196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.552198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.552326 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.552914 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:49.552921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:49.553181 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.553344 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.553697 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":167,"threads_started":1}
I20260620 14:33:49.554734 20313 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:33:49.555146 20313 tablet_bootstrap.cc:492] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap complete.
I20260620 14:33:49.556465 20313 ts_tablet_manager.cc:1397] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260620 14:33:49.557253 20421 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:49.557473 20421 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:49.557965 20421 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
I20260620 14:33:49.559514 20313 raft_consensus.cc:348] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:49.560056 20313 raft_consensus.cc:729] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2bb8b360044cb1b9afc6f72693e21c, State: Initialized, Role: FOLLOWER
I20260620 14:33:49.560492 20232 ts_manager.cc:194] Registered new tserver with Master: 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:49.560572 20313 consensus_queue.cc:260] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:49.561650 20313 ts_tablet_manager.cc:1428] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 14:33:49.564306 20232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:60771
I20260620 14:33:49.567050 20421 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.65@KRBTEST.COM
I20260620 14:33:49.575238 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.563685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.0.0.1:34668)
0620 14:33:49.564478 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.564531 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.564688 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.565304 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.565311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.565757 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:49.565894 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.565901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.567300 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.567304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.568254 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.568266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.568996 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.569000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.569125 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.569665 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:49.569679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:49.571274 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:49.573504 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.573514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.573519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.573780 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.574097 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.574099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.574100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.574173 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.574672 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:49.574677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:49.574810 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.574922 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.575095 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":112,"threads_started":1}
I20260620 14:33:49.575381 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.563784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.4.58.65:44939)
0620 14:33:49.564195 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.564206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.564638 (+ 432us) server_negotiation.cc:407] Connection header received
0620 14:33:49.564873 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.564879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.565109 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.565256 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.566027 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.567064 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.568384 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.568882 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.568966 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.571500 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.571512 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.571550 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.571581 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.573384 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.573897 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.573903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.573905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.574004 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.574247 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.574250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.574252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.574569 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.574692 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.575023 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.575215 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":95,"threads_started":1}
I20260620 14:33:49.576452 20355 tablet_service.cc:1514] Processing DeleteTablet for tablet db12d8bd18154f11afd632175debabb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:29 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:34668
I20260620 14:33:49.577347 20430 tablet_replica.cc:330] stopping tablet replica
I20260620 14:33:49.577627 20430 raft_consensus.cc:2229] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:49.577877 20430 raft_consensus.cc:2258] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:49.578816 20430 ts_tablet_manager.cc:1905] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:49.582559 20430 ts_tablet_manager.cc:1918] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:49.582656 20430 log.cc:1196] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal/wals/db12d8bd18154f11afd632175debabb0
I20260620 14:33:49.582878 20430 ts_tablet_manager.cc:1939] T db12d8bd18154f11afd632175debabb0 P 2e2bb8b360044cb1b9afc6f72693e21c: Deleting consensus metadata
I20260620 14:33:49.584626 20216 catalog_manager.cc:4841] TS 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939): tablet db12d8bd18154f11afd632175debabb0 (table test-table [id=e0f8e6b0aa774d3f98a6426452c2a49f]) successfully deleted
W20260620 14:33:49.663385 20425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:49.663666 20425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:49.663709 20425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:49.667768 20425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:49.667835 20425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:49.667894 20425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:33:49.673604 20425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:46079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=34681
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:49.674669 20425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:49.675652 20425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:49.685837 20438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:49.685868 20435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.685987 20425 server_base.cc:1034] running on GCE node
W20260620 14:33:49.685837 20436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.686619 20425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:49.687346 20425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:49.688545 20425 hybrid_clock.cc:648] HybridClock initialized: now 1781966029688510 us; error 55 us; skew 500 ppm
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:49.691336 20425 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:33:49.692292 20425 webserver.cc:460] Webserver started at http://127.4.58.66:34681/ using document root <none> and password file <none>
I20260620 14:33:49.692903 20425 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:49.692988 20425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:49.697095 20425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:33:49.699345 20445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.700596 20425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:33:49.700701 20425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "04b909af50314988a3286aebeaa10139"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "e3bf745fec5a6c4a866f9b5bb538e649"
server_key_iv: "00546f98832fcf17f390d6563ffea153"
server_key_version: "encryptionkey@0"
I20260620 14:33:49.701309 20425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:49.712960 20425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:49.715762 20425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:49.715996 20425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:49.716598 20425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:49.717442 20425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:49.717490 20425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.717532 20425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:49.717562 20425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.726503 20425 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46079
I20260620 14:33:49.726517 20558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46079 every 8 connection(s)
I20260620 14:33:49.727511 20425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:33:49.729923 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20425
I20260620 14:33:49.730387 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:40645
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=46187
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:49.741829 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.729271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:38373 (local address 127.4.58.126:39907)
0620 14:33:49.729450 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.729454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.730585 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 14:33:49.731560 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.731564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.731612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.731686 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.733268 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.733791 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.735480 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.735708 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.735788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.738499 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.738502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.738504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.738532 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.739846 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.740324 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.740327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.740328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.740405 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.740711 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.740715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.740716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.740939 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.741026 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.741522 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.741659 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:33:49.742684 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.729685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:38373)
0620 14:33:49.730421 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.730470 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.731318 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.731900 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.731909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.732392 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:49.733116 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.733136 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.733911 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.733915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.735360 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.735372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.735862 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.735867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.736070 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.736669 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:49.736693 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:49.738328 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:49.739956 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.739962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.739979 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.740229 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.740487 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.740490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.740492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.740624 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.741035 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:49.741041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:49.741293 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.741463 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.741801 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":129,"threads_started":1}
I20260620 14:33:49.743831 20559 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:49.744061 20559 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:49.744572 20559 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
I20260620 14:33:49.745666 20232 ts_manager.cc:194] Registered new tserver with Master: 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079)
I20260620 14:33:49.746213 20232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:38373
W20260620 14:33:49.843014 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:33:49.843286 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:33:49.843329 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:33:49.847383 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:33:49.847450 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:33:49.847515 20562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:33:49.852873 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:40645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=46187
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:33:49.853922 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:33:49.854909 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:33:49.864741 20568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:33:49.864737 20567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.864830 20562 server_base.cc:1034] running on GCE node
W20260620 14:33:49.864741 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:33:49.865430 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:33:49.866086 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:33:49.867249 20562 hybrid_clock.cc:648] HybridClock initialized: now 1781966029867222 us; error 41 us; skew 500 ppm
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:33:49.870163 20562 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:33:49.871004 20562 webserver.cc:460] Webserver started at http://127.4.58.67:46187/ using document root <none> and password file <none>
I20260620 14:33:49.871622 20562 fs_manager.cc:359] Metadata directory not provided
I20260620 14:33:49.871697 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:33:49.875633 20562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:49.877926 20577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.879429 20562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:33:49.879539 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "5a48e778682740cabd253bd8c211cbb6"
server_key_iv: "13cdd5f8c78fb2fc78ad6373abc7d9de"
server_key_version: "encryptionkey@0"
I20260620 14:33:49.880165 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:33:49.915164 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:33:49.918473 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:33:49.918655 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:33:49.919275 20562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:33:49.920106 20562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:33:49.920152 20562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.920197 20562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:33:49.920225 20562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:33:49.928648 20562 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40645
I20260620 14:33:49.928699 20690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40645 every 8 connection(s)
I20260620 14:33:49.929654 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:33:49.937448 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20562
Jun 20 14:33:49 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966029, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:33:49.943696 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.931491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:49759 (local address 127.4.58.126:39907)
0620 14:33:49.931628 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.931632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.932669 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 14:33:49.933633 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.933637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.933695 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.933792 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.934988 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.935508 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.936653 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.936934 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.937023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.939660 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.939665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.939667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.939694 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.941272 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.941911 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.941914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.941916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.942095 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.942548 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.942557 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.942558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.942740 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.942842 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.943380 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.943514 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:33:49.944739 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.931862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:49759)
0620 14:33:49.932517 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.932558 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.933418 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.933981 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.933988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.934363 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:33:49.934844 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.934858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.935628 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.935631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.936533 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.936542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.937090 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.937094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.937307 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.937955 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:33:49.937977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:33:49.939513 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:33:49.941391 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.941398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.941414 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.941787 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.942237 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:33:49.942240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:33:49.942241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:33:49.942427 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:33:49.942854 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:33:49.942860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:33:49.943133 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.943319 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.943677 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":115,"threads_started":1}
I20260620 14:33:49.945932 20691 heartbeater.cc:347] Connected to a master server at 127.4.58.126:39907
I20260620 14:33:49.946166 20691 heartbeater.cc:464] Registering TS with master...
I20260620 14:33:49.946686 20691 heartbeater.cc:511] Master 127.4.58.126:39907 requested a full tablet report, sending...
I20260620 14:33:49.947196 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.938103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.4.58.126:39907)
0620 14:33:49.938350 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.938354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.938524 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:33:49.939188 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.939192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.939243 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.939321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.940092 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.940832 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.941982 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.942323 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.942405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.943715 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.943719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.943721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.943742 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.945603 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.946022 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.946025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.946027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.946113 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.946378 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.946381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.946383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.946557 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.946749 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.946930 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.947046 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260620 14:33:49.947729 20229 ts_manager.cc:194] Registered new tserver with Master: 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
I20260620 14:33:49.948465 20229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:49759
I20260620 14:33:49.948885 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:33:49.952611 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.949489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.4.58.126:39907)
0620 14:33:49.949604 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.949608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.949818 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:33:49.949959 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.949961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.950051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.950130 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:49.950361 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.950841 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.951672 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.951892 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.951965 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.952186 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:49.952319 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:33:49.952512 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41064: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:33:49.952685 20695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41064: token expired
W20260620 14:33:49.953099 20162 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.58.126:39907)
I20260620 14:33:49.959503 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.953445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.4.58.126:39907)
0620 14:33:49.953549 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.953552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.953563 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:33:49.953636 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.953638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.953670 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.953728 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:33:49.954271 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.954723 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.955477 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.955670 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.955736 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.956960 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:33:49.956963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:33:49.956965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:33:49.956982 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:33:49.958112 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.958458 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.958460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.958461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.958526 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:33:49.958793 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:33:49.958795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:33:49.958796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:33:49.958948 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:33:49.959040 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.959206 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.959339 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:33:49.961841 20229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:33:49.965178 20229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:33:49.974936 20355 tablet_service.cc:1467] Processing CreateTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 (DEFAULT_TABLE table=test-table [id=d11f5611c19143a7ae0ef1767155ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:49.975641 20355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60dc93dbc38440ebefb1e6f1039a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:49.978361 20313 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap starting.
I20260620 14:33:49.979278 20313 tablet_bootstrap.cc:654] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:49.980158 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.973760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40645 (local address 127.0.0.1:48262)
0620 14:33:49.973895 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.973935 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.974045 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.975696 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.975700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.975720 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:49.975836 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.975842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.977491 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.977494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.978692 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.978702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.979747 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.979751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.979844 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.979891 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.979960 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.980040 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:33:49.980392 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.973864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46079 (local address 127.0.0.1:46640)
0620 14:33:49.974509 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:49.974527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:49.974624 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:49.976131 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:49.976134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:49.976157 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:49.976242 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.976247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.977781 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.977785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.978796 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:49.978808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.980024 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.980027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:49.980122 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.980147 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:33:49.980196 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:33:49.980263 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":111,"threads_started":1}
I20260620 14:33:49.980674 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.974773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.4.58.67:40645)
0620 14:33:49.975127 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.975139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.975165 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:33:49.975249 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.975255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.975505 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.975988 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:49.976159 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.977367 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.978826 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.979632 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.979739 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.979893 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.980428 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.980538 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":154,"threads_started":1}
I20260620 14:33:49.981030 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:49.975172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.4.58.66:46079)
0620 14:33:49.975502 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:49.975513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:49.975537 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:33:49.975613 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:49.975619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:49.975965 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:49.976382 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:49.976552 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.977628 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.978935 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:49.979867 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:49.980555 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:49.980714 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:49.980796 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:33:49.980890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
I20260620 14:33:49.981233 20313 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: No bootstrap required, opened a new log
I20260620 14:33:49.981329 20313 ts_tablet_manager.cc:1397] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:33:49.981787 20313 raft_consensus.cc:348] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:49.981909 20313 raft_consensus.cc:374] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:49.981948 20313 raft_consensus.cc:729] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2bb8b360044cb1b9afc6f72693e21c, State: Initialized, Role: FOLLOWER
I20260620 14:33:49.982134 20313 consensus_queue.cc:260] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:49.982371 20313 ts_tablet_manager.cc:1428] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:33:49.983784 20493 tablet_service.cc:1467] Processing CreateTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 (DEFAULT_TABLE table=test-table [id=d11f5611c19143a7ae0ef1767155ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:49.983794 20625 tablet_service.cc:1467] Processing CreateTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 (DEFAULT_TABLE table=test-table [id=d11f5611c19143a7ae0ef1767155ec4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:33:49.984985 20625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60dc93dbc38440ebefb1e6f1039a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:49.985172 20493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60dc93dbc38440ebefb1e6f1039a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:33:49.993466 20704 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Bootstrap starting.
I20260620 14:33:49.993466 20705 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Bootstrap starting.
I20260620 14:33:49.996380 20705 tablet_bootstrap.cc:654] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:49.996380 20704 tablet_bootstrap.cc:654] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Neither blocks nor log segments found. Creating new log.
I20260620 14:33:49.997392 20705 log.cc:824] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:49.997392 20704 log.cc:824] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:33:49.999424 20705 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: No bootstrap required, opened a new log
I20260620 14:33:49.999436 20704 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: No bootstrap required, opened a new log
I20260620 14:33:49.999616 20705 ts_tablet_manager.cc:1397] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:33:49.999617 20704 ts_tablet_manager.cc:1397] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:33:50.002530 20704 raft_consensus.cc:348] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.002530 20705 raft_consensus.cc:348] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.002805 20704 raft_consensus.cc:374] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:50.002841 20705 raft_consensus.cc:374] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:33:50.002864 20704 raft_consensus.cc:729] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a41710e3d0e4daeb4e7d58ca12fa1b8, State: Initialized, Role: FOLLOWER
I20260620 14:33:50.002892 20705 raft_consensus.cc:729] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b909af50314988a3286aebeaa10139, State: Initialized, Role: FOLLOWER
I20260620 14:33:50.003329 20704 consensus_queue.cc:260] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.003523 20705 consensus_queue.cc:260] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.004078 20691 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
I20260620 14:33:50.004401 20704 ts_tablet_manager.cc:1428] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 14:33:50.004513 20705 ts_tablet_manager.cc:1428] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:33:50.004906 20559 heartbeater.cc:503] Master 127.4.58.126:39907 was elected leader, sending a full tablet report...
W20260620 14:33:50.038048 20422 tablet.cc:2369] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:50.155977 20708 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:33:50.156190 20708 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.157397 20708 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:50.162586 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.157849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:57217 (local address 127.4.58.66:46079)
0620 14:33:50.157983 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:50.157988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:50.158328 (+ 340us) server_negotiation.cc:407] Connection header received
0620 14:33:50.158401 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:50.158404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:50.158454 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:50.158523 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:50.158732 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.159542 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.161263 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.161913 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.162033 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.162103 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:50.162363 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:33:50.162443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:33:50.162588 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.157850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:56017 (local address 127.4.58.65:44939)
0620 14:33:50.158165 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:50.158169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:50.158184 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:33:50.158235 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:50.158237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:50.158331 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:50.158453 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:33:50.158733 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.159541 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.161236 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.161940 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.162034 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.162204 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:50.162371 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:33:50.162445 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260620 14:33:50.162640 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.157699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.4.58.67:56017)
0620 14:33:50.157881 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:50.157897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:50.158012 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:50.158474 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:50.158476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:50.158490 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:50.158615 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:50.158622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.159650 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.159654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:50.161111 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:50.161117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.162051 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.162055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:50.162219 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.162289 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:33:50.162350 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:33:50.162456 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":134}
I20260620 14:33:50.162652 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.157699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46079 (local address 127.4.58.67:57217)
0620 14:33:50.158230 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:50.158242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:33:50.158328 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:33:50.158543 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:33:50.158546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:33:50.158558 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:33:50.158641 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:50.158645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.159650 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.159654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:50.161102 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:33:50.161116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.162051 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.162055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:33:50.162202 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.162255 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:33:50.162336 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:33:50.162444 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":326,"thread_start_us":111,"threads_started":1}
I20260620 14:33:50.163335 20374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60dc93dbc38440ebefb1e6f1039a233" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" is_pre_election: true
I20260620 14:33:50.163657 20374 raft_consensus.cc:2454] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 0.
I20260620 14:33:50.163841 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60dc93dbc38440ebefb1e6f1039a233" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b909af50314988a3286aebeaa10139" is_pre_election: true
I20260620 14:33:50.164177 20513 raft_consensus.cc:2454] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 0.
I20260620 14:33:50.164214 20578 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a41710e3d0e4daeb4e7d58ca12fa1b8, 2e2bb8b360044cb1b9afc6f72693e21c; no voters:
I20260620 14:33:50.164541 20708 raft_consensus.cc:2781] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:33:50.164678 20708 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:33:50.164739 20708 raft_consensus.cc:3037] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:50.166097 20708 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.166479 20708 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 election: Requested vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939)
I20260620 14:33:50.166865 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60dc93dbc38440ebefb1e6f1039a233" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b909af50314988a3286aebeaa10139"
I20260620 14:33:50.167006 20513 raft_consensus.cc:3037] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:50.167642 20374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60dc93dbc38440ebefb1e6f1039a233" candidate_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
I20260620 14:33:50.167786 20374 raft_consensus.cc:3037] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:33:50.168363 20513 raft_consensus.cc:2454] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 1.
I20260620 14:33:50.168784 20581 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8; no voters:
I20260620 14:33:50.168970 20708 raft_consensus.cc:2781] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:33:50.169044 20374 raft_consensus.cc:2454] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41710e3d0e4daeb4e7d58ca12fa1b8 in term 1.
I20260620 14:33:50.169173 20708 raft_consensus.cc:686] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 LEADER]: Becoming Leader. State: Replica: 2a41710e3d0e4daeb4e7d58ca12fa1b8, State: Running, Role: LEADER
I20260620 14:33:50.169716 20708 consensus_queue.cc:237] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:33:50.173092 20229 catalog_manager.cc:5495] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } health_report { overall_health: UNKNOWN } } }
W20260620 14:33:50.180516 20692 tablet.cc:2369] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:50.223156 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.219810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.4.58.67:40645)
0620 14:33:50.219931 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:33:50.219935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:33:50.219947 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:33:50.220022 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:33:50.220024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:33:50.220081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:33:50.220173 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:33:50.220420 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.220977 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.221795 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:33:50.222018 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:33:50.222095 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:33:50.222356 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:33:50.222809 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:33:50.222902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:33:50.222965 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:33:50.223018 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:33:50.229012 20374 raft_consensus.cc:1261] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Refusing update from remote peer 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:50.229015 20513 raft_consensus.cc:1261] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Refusing update from remote peer 2a41710e3d0e4daeb4e7d58ca12fa1b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:33:50.229715 20712 consensus_queue.cc:1035] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:33:50.229898 20708 consensus_queue.cc:1035] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:33:50.230453 20560 tablet.cc:2369] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:33:50.235962 20714 mvcc.cc:205] Tried to move back new op lower bound from 7298932859810672640 to 7298932859578175488. Current Snapshot: MvccSnapshot[applied={T|T < 7298932859810672640}]
I20260620 14:33:50.236515 20715 mvcc.cc:205] Tried to move back new op lower bound from 7298932859810672640 to 7298932859578175488. Current Snapshot: MvccSnapshot[applied={T|T < 7298932859810672640}]
I20260620 14:33:50.245048 20232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:50.245203 20232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:33:50.247613 20232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Sending DeleteTablet for 3 replicas of tablet a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:33:50.248348 20493 tablet_service.cc:1514] Processing DeleteTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:50 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:46640
I20260620 14:33:50.248348 20625 tablet_service.cc:1514] Processing DeleteTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:50 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:48262
I20260620 14:33:50.248502 20355 tablet_service.cc:1514] Processing DeleteTablet for tablet a60dc93dbc38440ebefb1e6f1039a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:33:50 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:34668
I20260620 14:33:50.248919 20725 tablet_replica.cc:330] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: stopping tablet replica
I20260620 14:33:50.249022 20726 tablet_replica.cc:330] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: stopping tablet replica
I20260620 14:33:50.249147 20726 raft_consensus.cc:2229] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:50.249205 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20270
I20260620 14:33:50.249300 20726 raft_consensus.cc:2258] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:50.249466 20725 raft_consensus.cc:2229] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:33:50.249851 20725 raft_consensus.cc:2258] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:50.250207 20724 tablet_replica.cc:330] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: stopping tablet replica
I20260620 14:33:50.250496 20724 raft_consensus.cc:2229] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:33:50.250839 20724 raft_consensus.cc:2258] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:33:50.251772 20725 ts_tablet_manager.cc:1905] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:50.252270 20724 ts_tablet_manager.cc:1905] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:33:50.254738 20725 ts_tablet_manager.cc:1918] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:50.254738 20724 ts_tablet_manager.cc:1918] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:33:50.254840 20724 log.cc:1196] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:33:50.254971 20725 log.cc:1196] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:33:50.255154 20724 ts_tablet_manager.cc:1939] T a60dc93dbc38440ebefb1e6f1039a233 P 04b909af50314988a3286aebeaa10139: Deleting consensus metadata
I20260620 14:33:50.255412 20725 ts_tablet_manager.cc:1939] T a60dc93dbc38440ebefb1e6f1039a233 P 2a41710e3d0e4daeb4e7d58ca12fa1b8: Deleting consensus metadata
I20260620 14:33:50.256506 20219 catalog_manager.cc:4841] TS 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): tablet a60dc93dbc38440ebefb1e6f1039a233 (table test-table [id=d11f5611c19143a7ae0ef1767155ec4a]) successfully deleted
I20260620 14:33:50.256506 20217 catalog_manager.cc:4841] TS 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): tablet a60dc93dbc38440ebefb1e6f1039a233 (table test-table [id=d11f5611c19143a7ae0ef1767155ec4a]) successfully deleted
W20260620 14:33:50.257843 20216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:44939 (error 108)
I20260620 14:33:50.258059 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20425
I20260620 14:33:50.259406 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:33:50.259079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:44939 (local address 127.0.0.1:34672)
0620 14:33:50.259206 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:33:50.259329 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:33:50.259622 20216 catalog_manager.cc:4584] TS 2e2bb8b360044cb1b9afc6f72693e21c (127.4.58.65:44939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a60dc93dbc38440ebefb1e6f1039a233: Network error: Client connection negotiation failed: client connection to 127.4.58.65:44939: connect: Connection refused (error 111)
I20260620 14:33:50.264822 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20562
I20260620 14:33:50.270174 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20199
I20260620 14:34:10.276204 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--webserver_interface=127.4.58.126
--webserver_port=46771
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:39907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:10.385790 20728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.386051 20728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.386096 20728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.390179 20728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:34:10.390244 20728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:34:10.390259 20728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:34:10.390270 20728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:10.390287 20728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.390305 20728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.390333 20728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:10.390359 20728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:10.395746 20728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:39907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:39907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:10.396868 20728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.397950 20728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:10.407248 20728 server_base.cc:1034] running on GCE node
W20260620 14:34:10.407203 20733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.407203 20734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.407208 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.408154 20728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.409363 20728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.410542 20728 hybrid_clock.cc:648] HybridClock initialized: now 1781966050410519 us; error 38 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.417214 20728 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:10.418267 20728 webserver.cc:460] Webserver started at http://127.4.58.126:46771/ using document root <none> and password file <none>
I20260620 14:34:10.418869 20728 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.418943 20728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.423038 20728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:10.425261 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.426651 20728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:10.426806 20728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "aa91e24392844917ae3845b31c4ab028"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "3a50fcd6515336a3569c418ef408f075"
server_key_iv: "06016e27b33e05d24c4b6cb95c387bfb"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.427438 20728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.438457 20728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.441687 20728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.441885 20728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.448833 20728 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:39907
I20260620 14:34:10.449091 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:39907 every 8 connection(s)
I20260620 14:34:10.449949 20728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:10.452688 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20728
I20260620 14:34:10.453123 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:44939
--local_ip_for_outbound_sockets=127.4.58.65
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=41593
--webserver_interface=127.4.58.65
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:10.454412 20796 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:34:10.456350 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap starting.
I20260620 14:34:10.473810 20796 log.cc:824] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:10.484974 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:10.485359 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Bootstrap complete.
I20260620 14:34:10.489183 20796 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:34:10.489580 20796 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Initialized, Role: FOLLOWER
I20260620 14:34:10.490070 20796 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:34:10.490198 20796 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:10.490262 20796 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:10.490353 20796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:34:10.492249 20796 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:34:10.492560 20796 leader_election.cc:302] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa91e24392844917ae3845b31c4ab028; no voters:
I20260620 14:34:10.492897 20796 leader_election.cc:288] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:34:10.492997 20800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:34:10.493188 20800 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [term 3 LEADER]: Becoming Leader. State: Replica: aa91e24392844917ae3845b31c4ab028, State: Running, Role: LEADER
I20260620 14:34:10.493541 20800 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } }
I20260620 14:34:10.494159 20796 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:10.495036 20802 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa91e24392844917ae3845b31c4ab028. Latest consensus state: current_term: 3 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:34:10.495208 20802 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:10.495613 20809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:10.495637 20801 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aa91e24392844917ae3845b31c4ab028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa91e24392844917ae3845b31c4ab028" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 39907 } } }
I20260620 14:34:10.495760 20801 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:10.501360 20809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:10.502148 20809 catalog_manager.cc:1259] Loaded cluster ID: 447298d9ab84451fa928e79531796031
I20260620 14:34:10.502275 20809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:10.504937 20809 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:10.505781 20809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Loaded TSK: 0
I20260620 14:34:10.505961 20809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa91e24392844917ae3845b31c4ab028: Loaded TSK: 1
F20260620 14:34:10.506837 20809 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08d9d549ad google::LogMessage::Fail() at ??:0
@ 0x7f08d9d5788a google::LogMessage::SendToLog() at ??:0
@ 0x7f08d9d544de google::LogMessage::Flush() at ??:0
@ 0x7f08d9d55a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f08dca1818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f08dca10f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f08e46b97bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f08e469399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f08e4694178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f08e46d5df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f08e46f482b std::function<>::operator()() at ??:0
@ 0x7f08e4693b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f08e4694f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f08e4691579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f08e46d5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f08dbfa63d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f08dbfa6c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f08dbfa83f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f08dbf97512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f08dd5db6db start_thread at ??:0
@ 0x7f08d885261f clone at ??:0
W20260620 14:34:10.583616 20798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.583885 20798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.583933 20798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.588399 20798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.588450 20798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.588510 20798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:10.594244 20798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:44939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=41593
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:10.595299 20798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.596436 20798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:10.607470 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.607378 20843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.607631 20798 server_base.cc:1034] running on GCE node
W20260620 14:34:10.607479 20844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.608359 20798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.609230 20798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.610440 20798 hybrid_clock.cc:648] HybridClock initialized: now 1781966050610411 us; error 33 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.616662 20798 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:34:10.617853 20798 webserver.cc:460] Webserver started at http://127.4.58.65:41593/ using document root <none> and password file <none>
I20260620 14:34:10.618618 20798 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.618718 20798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.623936 20798 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:34:10.626694 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.628257 20798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:34:10.628427 20798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "2e2bb8b360044cb1b9afc6f72693e21c"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "168390242d8fe3c68f662ee936cbf3cc"
server_key_iv: "44a9557a3f1c17897c3a5348ea7299fe"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.629187 20798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.661281 20798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.665594 20798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.665792 20798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.666538 20798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:10.667932 20860 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:34:10.673913 20798 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:34:10.674090 20798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 14:34:10.674605 20798 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:34:10.677103 20798 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:34:10.677263 20798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:10.677516 20860 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap starting.
I20260620 14:34:10.688446 20798 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:44939
I20260620 14:34:10.688480 20967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:44939 every 8 connection(s)
I20260620 14:34:10.689949 20798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:10.690598 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20798
I20260620 14:34:10.691061 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:46079
--local_ip_for_outbound_sockets=127.4.58.66
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=34681
--webserver_interface=127.4.58.66
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:10.699271 20860 log.cc:824] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:10.707551 20860 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:34:10.707918 20860 tablet_bootstrap.cc:492] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Bootstrap complete.
I20260620 14:34:10.708889 20860 ts_tablet_manager.cc:1397] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.012s
I20260620 14:34:10.711308 20860 raft_consensus.cc:348] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:10.711722 20860 raft_consensus.cc:729] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2bb8b360044cb1b9afc6f72693e21c, State: Initialized, Role: FOLLOWER
I20260620 14:34:10.712186 20860 consensus_queue.cc:260] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:10.713158 20860 ts_tablet_manager.cc:1428] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260620 14:34:10.771492 20971 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:34:10.699013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:35643)
0620 14:34:10.700421 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.700467 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:10.701442 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:10.770709 (+ 69267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":156,"threads_started":1}
W20260620 14:34:10.772179 20969 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:39907 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:39907: Client connection negotiation failed: client connection to 127.4.58.126:39907: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:34:10.772979 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.772669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:41291)
0620 14:34:10.772826 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.772904 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:34:10.773675 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.773458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:59727)
0620 14:34:10.773559 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.773608 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:34:10.773895 20969 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:34:10.825081 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:10.825366 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:10.825412 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:10.829845 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:10.829910 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:10.829972 20968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:10.835350 20968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:46079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=34681
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:10.836439 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:10.837486 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:10.847505 20968 server_base.cc:1034] running on GCE node
W20260620 14:34:10.847446 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.847440 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:10.847442 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:10.848176 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:10.848899 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:10.850085 20968 hybrid_clock.cc:648] HybridClock initialized: now 1781966050850062 us; error 35 us; skew 500 ppm
Jun 20 14:34:10 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:10.852881 20968 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:34:10.853806 20968 webserver.cc:460] Webserver started at http://127.4.58.66:34681/ using document root <none> and password file <none>
I20260620 14:34:10.854351 20968 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:10.854418 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:10.858304 20968 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:10.860409 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.861774 20968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:10.861892 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "04b909af50314988a3286aebeaa10139"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "e3bf745fec5a6c4a866f9b5bb538e649"
server_key_iv: "00546f98832fcf17f390d6563ffea153"
server_key_version: "encryptionkey@0"
I20260620 14:34:10.862459 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:10.872470 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:10.875742 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:10.875952 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:10.876545 20968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:10.877388 20968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:10.877436 20968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.877487 20968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:10.877513 20968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:10.886296 20968 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46079
I20260620 14:34:10.886317 21101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46079 every 8 connection(s)
I20260620 14:34:10.887327 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:10.888957 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 20968
I20260620 14:34:10.889446 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:40645
--local_ip_for_outbound_sockets=127.4.58.67
--tserver_master_addrs=127.4.58.126:39907
--webserver_port=46187
--webserver_interface=127.4.58.67
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:34:10.892092 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.889821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:43007)
0620 14:34:10.890357 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.890473 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":129,"threads_started":1}
W20260620 14:34:10.892573 21102 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:39907 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:39907: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
I20260620 14:34:10.893188 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.892976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:43135)
0620 14:34:10.893086 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.893136 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:34:10.893808 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:10.893610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:41113)
0620 14:34:10.893711 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:10.893755 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:34:10.893988 21102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:34:11.001931 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:11.002207 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:11.002251 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:11.006428 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:11.006495 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:11.006564 21105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:11.011955 21105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:40645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=46187
--tserver_master_addrs=127.4.58.126:39907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:11.013077 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:11.014117 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:11.023679 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:11.023741 21105 server_base.cc:1034] running on GCE node
W20260620 14:34:11.023679 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:11.023684 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:11.024307 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:11.024976 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:11.026125 21105 hybrid_clock.cc:648] HybridClock initialized: now 1781966051026105 us; error 32 us; skew 500 ppm
Jun 20 14:34:11 dist-test-slave-gqzk krb5kdc[19652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966051, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:11.028733 21105 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:34:11.029654 21105 webserver.cc:460] Webserver started at http://127.4.58.67:46187/ using document root <none> and password file <none>
I20260620 14:34:11.030200 21105 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:11.030262 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:11.034116 21105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:34:11.036276 21120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:11.037611 21105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:11.037717 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8"
format_stamp: "Formatted at 2026-06-20 14:33:28 on dist-test-slave-gqzk"
server_key: "5a48e778682740cabd253bd8c211cbb6"
server_key_iv: "13cdd5f8c78fb2fc78ad6373abc7d9de"
server_key_version: "encryptionkey@0"
I20260620 14:34:11.038281 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:11.047572 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:11.050653 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:11.050838 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:11.051405 21105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:11.052222 21105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:11.052268 21105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:11.052310 21105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:11.052338 21105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:11.060954 21105 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40645
I20260620 14:34:11.061031 21233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40645 every 8 connection(s)
I20260620 14:34:11.061985 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:11.065816 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.064168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:48255)
0620 14:34:11.064891 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.065004 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":125,"threads_started":1}
W20260620 14:34:11.066305 21234 heartbeater.cc:644] Failed to heartbeat to 127.4.58.126:39907 (0 consecutive failures): Network error: Failed to ping master at 127.4.58.126:39907: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
I20260620 14:34:11.066846 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21105
I20260620 14:34:11.066978 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.066760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:58903)
0620 14:34:11.066862 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.066913 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:34:11.067732 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.067542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:59599)
0620 14:34:11.067645 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.067688 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:34:11.067896 21234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:34:11.774967 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.774505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:35587)
0620 14:34:11.774799 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.774861 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260620 14:34:11.895025 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:11.894605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:52281)
0620 14:34:11.894862 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:11.894921 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260620 14:34:12.068854 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.068430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:60629)
0620 14:34:12.068653 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:12.068709 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260620 14:34:12.116894 21241 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:12.117085 21241 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:12.118296 21241 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
Jun 20 14:34:12 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.66@KRBTEST.COM
Jun 20 14:34:12 dist-test-slave-gqzk krb5kdc[19652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966050, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.67@KRBTEST.COM
I20260620 14:34:12.132675 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.118523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46079 (local address 127.4.58.65:55785)
0620 14:34:12.118710 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:12.118724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:12.118824 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:12.120712 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:12.120718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:12.121277 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:12.121781 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:12.121796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.123109 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.123112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:12.124097 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:12.124107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.124907 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:12.125109 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:12.125795 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:12.125816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:12.127400 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:12.129977 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:12.129982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:12.129993 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:12.130310 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:12.130638 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:12.130643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:12.130806 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:12.131548 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:12.131553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:12.131868 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 14:34:12.132088 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 14:34:12.132477 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":211,"spinlock_wait_cycles":6656}
I20260620 14:34:12.132694 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.118968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:55785 (local address 127.4.58.66:46079)
0620 14:34:12.119356 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:12.119378 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:12.119413 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:34:12.120044 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:12.120049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:12.120350 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:12.120627 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:12.121947 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.122981 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124236 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.124786 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124966 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:12.127626 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:12.127631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:12.127645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:12.127675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:12.129778 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130417 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:12.130422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:12.130426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:12.130538 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130893 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:12.130897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:12.130899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:12.131417 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:12.131549 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:12.132214 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 14:34:12.132510 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260620 14:34:12.132694 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.118746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:54521 (local address 127.4.58.67:40645)
0620 14:34:12.119085 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:12.119110 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:12.119159 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:34:12.119988 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:12.119994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:12.120350 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:12.120627 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:12.121954 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.122942 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124286 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.124741 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124951 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:12.127854 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:12.127860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:12.127876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:12.127910 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:12.129783 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130725 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:12.130730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:12.130733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:12.130850 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:12.131111 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:12.131114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:12.131116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:12.131549 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:12.131654 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:12.132214 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 14:34:12.132510 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260620 14:34:12.132675 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.118523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40645 (local address 127.4.58.65:54521)
0620 14:34:12.119076 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:12.119090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:12.119222 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:12.120719 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:12.120723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:12.121278 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:12.121781 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:12.121796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.123107 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.123110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:12.124137 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:12.124148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:12.124904 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:12.124908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:12.125112 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:12.125816 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:12.125830 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:12.127637 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:12.130422 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:12.130425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:12.130618 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:12.130940 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:12.130943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:12.130944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:12.131018 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:12.131656 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:12.131658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:12.131857 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 14:34:12.132088 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 14:34:12.132537 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":456,"spinlock_wait_cycles":62464,"thread_start_us":82,"threads_started":1}
W20260620 14:34:12.135851 20855 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
W20260620 14:34:12.136231 20857 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:34:12.136310 20857 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2bb8b360044cb1b9afc6f72693e21c; no voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8
I20260620 14:34:12.136571 21241 raft_consensus.cc:2726] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:12.776417 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.775879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:49503)
0620 14:34:12.776236 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:12.776313 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":83,"threads_started":1}
I20260620 14:34:12.896337 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:12.895874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:46969)
0620 14:34:12.896148 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:12.896219 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260620 14:34:13.069902 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:13.069490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:37679)
0620 14:34:13.069743 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:13.069805 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260620 14:34:13.777782 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:13.777263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:33297)
0620 14:34:13.777606 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:13.777676 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260620 14:34:13.841646 21249 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:13.841812 21249 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:13.842370 21249 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
W20260620 14:34:13.843154 20855 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
W20260620 14:34:13.843278 20857 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:34:13.843348 20857 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2bb8b360044cb1b9afc6f72693e21c; no voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8
I20260620 14:34:13.843492 21249 raft_consensus.cc:2726] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:13.897572 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:13.897117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:46757)
0620 14:34:13.897388 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:13.897461 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260620 14:34:14.071065 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:14.070623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:51341)
0620 14:34:14.070897 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:14.070956 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260620 14:34:14.779264 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:14.778757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:47875)
0620 14:34:14.779086 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:14.779158 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260620 14:34:14.898989 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:14.898521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:38275)
0620 14:34:14.898806 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:14.898871 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260620 14:34:15.072250 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:15.071838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:36869)
0620 14:34:15.072094 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:15.072154 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260620 14:34:15.780741 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:15.780289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:59059)
0620 14:34:15.780573 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:15.780633 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260620 14:34:15.900211 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:15.899767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:49565)
0620 14:34:15.900017 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:15.900073 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260620 14:34:16.073504 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:16.073056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:50259)
0620 14:34:16.073339 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:16.073403 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260620 14:34:16.400364 21258 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:16.400547 21258 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:16.401093 21258 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
W20260620 14:34:16.401892 20855 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
W20260620 14:34:16.402024 20857 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:34:16.402122 20857 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2bb8b360044cb1b9afc6f72693e21c; no voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8
I20260620 14:34:16.402316 21258 raft_consensus.cc:2726] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:16.782068 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:16.781628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:53651)
0620 14:34:16.781908 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:16.781971 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260620 14:34:16.901451 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:16.900974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:50697)
0620 14:34:16.901291 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:16.901353 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260620 14:34:17.074611 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:17.074193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:57883)
0620 14:34:17.074463 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:17.074515 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260620 14:34:17.783353 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:17.782922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:41123)
0620 14:34:17.783187 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:17.783251 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260620 14:34:17.902827 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:17.902298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:36605)
0620 14:34:17.902658 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:17.902727 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":75,"threads_started":1}
I20260620 14:34:18.075857 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:18.075360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:44749)
0620 14:34:18.075690 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:18.075760 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":92,"threads_started":1}
I20260620 14:34:18.784804 21265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:18.784276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:60825)
0620 14:34:18.784580 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:18.784647 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260620 14:34:18.904175 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:18.903694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:56819)
0620 14:34:18.904012 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:18.904079 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260620 14:34:19.077191 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:19.076674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:59593)
0620 14:34:19.076974 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:19.077040 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
I20260620 14:34:19.786090 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:19.785624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:35003)
0620 14:34:19.785927 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:19.785983 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260620 14:34:19.905573 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:19.905120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:58855)
0620 14:34:19.905408 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:19.905475 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260620 14:34:20.078642 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:20.078106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:46851)
0620 14:34:20.078439 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:20.078514 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260620 14:34:20.354290 21271 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:20.354429 21271 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:20.354862 21271 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
W20260620 14:34:20.355670 20857 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
W20260620 14:34:20.355793 20855 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:34:20.355870 20855 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2bb8b360044cb1b9afc6f72693e21c; no voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8
I20260620 14:34:20.356017 21271 raft_consensus.cc:2726] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:20.787309 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:20.786905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:47557)
0620 14:34:20.787159 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:20.787213 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260620 14:34:20.906940 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:20.906460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:37375)
0620 14:34:20.906763 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:20.906840 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
I20260620 14:34:21.079921 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:21.079476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:33191)
0620 14:34:21.079743 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:21.079824 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260620 14:34:21.788424 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:21.788012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:59985)
0620 14:34:21.788250 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:21.788326 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260620 14:34:21.908213 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:21.907775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:43941)
0620 14:34:21.908065 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:21.908120 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
I20260620 14:34:22.081266 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:22.080735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:47459)
0620 14:34:22.081054 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:22.081130 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
I20260620 14:34:22.789628 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:22.789212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:43145)
0620 14:34:22.789467 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:22.789527 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260620 14:34:22.909502 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:22.909028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:41033)
0620 14:34:22.909345 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:22.909404 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
I20260620 14:34:23.082417 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:23.081978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:55707)
0620 14:34:23.082247 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:23.082306 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260620 14:34:23.303108 21281 raft_consensus.cc:482] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:23.303241 21281 raft_consensus.cc:504] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b909af50314988a3286aebeaa10139" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46079 } } peers { permanent_uuid: "2a41710e3d0e4daeb4e7d58ca12fa1b8" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40645 } } peers { permanent_uuid: "2e2bb8b360044cb1b9afc6f72693e21c" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 44939 } }
I20260620 14:34:23.303654 21281 leader_election.cc:288] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079), 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645)
W20260620 14:34:23.304466 20855 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2a41710e3d0e4daeb4e7d58ca12fa1b8 (127.4.58.67:40645): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
W20260620 14:34:23.304574 20857 leader_election.cc:341] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 04b909af50314988a3286aebeaa10139 (127.4.58.66:46079): Not found: Tablet not found: a60dc93dbc38440ebefb1e6f1039a233
I20260620 14:34:23.304661 20857 leader_election.cc:302] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2bb8b360044cb1b9afc6f72693e21c; no voters: 04b909af50314988a3286aebeaa10139, 2a41710e3d0e4daeb4e7d58ca12fa1b8
I20260620 14:34:23.304844 21281 raft_consensus.cc:2726] T a60dc93dbc38440ebefb1e6f1039a233 P 2e2bb8b360044cb1b9afc6f72693e21c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:34:23.790897 21282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:23.790460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:40885)
0620 14:34:23.790722 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:23.790786 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260620 14:34:23.910686 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:23.910283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:53225)
0620 14:34:23.910528 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:23.910589 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260620 14:34:24.083578 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:24.083117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:60881)
0620 14:34:24.083397 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:24.083457 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260620 14:34:24.792234 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:24.791742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:36753)
0620 14:34:24.792066 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:24.792135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260620 14:34:24.911976 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:24.911537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:60073)
0620 14:34:24.911813 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:24.911877 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260620 14:34:25.084775 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:25.084344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.67:41957)
0620 14:34:25.084593 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:25.084650 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260620 14:34:25.793841 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:25.793352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.65:56377)
0620 14:34:25.793649 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:25.793719 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260620 14:34:25.913362 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:25.912924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:39907 (local address 127.4.58.66:54213)
0620 14:34:25.913201 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:25.913264 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.126:39907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260620 14:34:26.067948 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20798
I20260620 14:34:26.072815 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 20968
I20260620 14:34:26.077630 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21105
2026-06-20T14:34:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59077 ms)
[----------] 30 tests from SecurityITest (241367 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:26 dist-test-slave-gqzk krb5kdc[21292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:26 dist-test-slave-gqzk krb5kdc[21292](info): set up 2 sockets
Jun 20 14:34:26 dist-test-slave-gqzk krb5kdc[21292](info): commencing operation
krb5kdc: starting...
W20260620 14:34:27.040454 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:27.161463 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33855
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:27.272444 21305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:27.272725 21305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:27.272790 21305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:27.276865 21305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:27.276927 21305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:27.276945 21305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:27.276961 21305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:27.276975 21305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:27.282569 21305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:27.283658 21305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:27.284677 21305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:27.294059 21313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:27.294060 21310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:27.294060 21311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:27.294409 21305 server_base.cc:1034] running on GCE node
I20260620 14:34:27.294950 21305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:27.296089 21305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:27.297281 21305 hybrid_clock.cc:648] HybridClock initialized: now 1781966067297253 us; error 39 us; skew 500 ppm
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:27.300031 21305 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:27.301076 21305 webserver.cc:460] Webserver started at http://127.4.58.126:35761/ using document root <none> and password file <none>
I20260620 14:34:27.301651 21305 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:27.301720 21305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:27.301913 21305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:27.303999 21305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "be1908d5e05343e2a5978e4af9fbbe56"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "47ac74ddb7a09a5875c9ef3c3beb6267"
server_key_iv: "02665720fdddd51f1841ee901563b449"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.304564 21305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "be1908d5e05343e2a5978e4af9fbbe56"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "47ac74ddb7a09a5875c9ef3c3beb6267"
server_key_iv: "02665720fdddd51f1841ee901563b449"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.308492 21305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:34:27.311053 21320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.312510 21305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:34:27.312661 21305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "be1908d5e05343e2a5978e4af9fbbe56"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "47ac74ddb7a09a5875c9ef3c3beb6267"
server_key_iv: "02665720fdddd51f1841ee901563b449"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.312805 21305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:27.327186 21305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:27.330387 21305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:27.330564 21305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:27.337308 21305 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:33855
I20260620 14:34:27.337327 21372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:33855 every 8 connection(s)
I20260620 14:34:27.338497 21305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:27.341406 21373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:27.346964 21373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: Bootstrap starting.
I20260620 14:34:27.347892 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21305
I20260620 14:34:27.348024 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:27.348292 4329 external_mini_cluster.cc:1428] Setting key 6d865ef79d8ab0725fe3c51611c1484d
I20260620 14:34:27.349838 21373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:27.350591 21373 log.cc:824] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:27.352824 21373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: No bootstrap required, opened a new log
I20260620 14:34:27.357136 21373 raft_consensus.cc:348] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } }
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:27.357383 21373 raft_consensus.cc:374] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:27.357452 21373 raft_consensus.cc:729] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1908d5e05343e2a5978e4af9fbbe56, State: Initialized, Role: FOLLOWER
I20260620 14:34:27.357896 21373 consensus_queue.cc:260] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } }
I20260620 14:34:27.358057 21373 raft_consensus.cc:388] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:27.358130 21373 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:27.358233 21373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:27.359356 21373 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } }
I20260620 14:34:27.359723 21373 leader_election.cc:302] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be1908d5e05343e2a5978e4af9fbbe56; no voters:
I20260620 14:34:27.360065 21373 leader_election.cc:288] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:27.360148 21378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:27.360316 21378 raft_consensus.cc:686] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [term 1 LEADER]: Becoming Leader. State: Replica: be1908d5e05343e2a5978e4af9fbbe56, State: Running, Role: LEADER
I20260620 14:34:27.360617 21378 consensus_queue.cc:237] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } }
I20260620 14:34:27.361418 21373 sys_catalog.cc:564] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:27.361804 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1908d5e05343e2a5978e4af9fbbe56. Latest consensus state: current_term: 1 leader_uuid: "be1908d5e05343e2a5978e4af9fbbe56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } } }
I20260620 14:34:27.361918 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:27.362582 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be1908d5e05343e2a5978e4af9fbbe56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1908d5e05343e2a5978e4af9fbbe56" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33855 } } }
I20260620 14:34:27.362694 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:27.362994 21382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:27.363303 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:27.349434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.4.58.126:33855)
0620 14:34:27.349856 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:27.349876 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:27.349919 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:34:27.350617 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:27.350636 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:27.351067 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:27.351449 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:27.352314 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.353432 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.354592 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.355093 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.355216 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:27.357973 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:27.357979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:27.357997 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:27.358031 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:27.359880 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.360350 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.360356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.360361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.360502 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.360905 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.360909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.360910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.361359 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:27.361492 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:27.361952 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 14:34:27.362263 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260620 14:34:27.366240 21382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:27.371989 21382 catalog_manager.cc:1347] Generated new cluster ID: 386758c72ffe45e0b8bec63b083d9ddc
I20260620 14:34:27.372054 21382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:27.391590 21382 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:27.393006 21382 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:27.399078 21382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: Generated new TSK 0
I20260620 14:34:27.399950 21382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:34:27.431495 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:27.543020 21399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:27.543294 21399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:27.543340 21399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:27.547304 21399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:27.547362 21399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:27.547420 21399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:27.552841 21399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:27.553880 21399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:27.554949 21399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:27.564685 21407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:27.564739 21404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:27.564688 21405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:27.564725 21399 server_base.cc:1034] running on GCE node
I20260620 14:34:27.565330 21399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:27.565999 21399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:27.567158 21399 hybrid_clock.cc:648] HybridClock initialized: now 1781966067567118 us; error 51 us; skew 500 ppm
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:27.569968 21399 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:34:27.570863 21399 webserver.cc:460] Webserver started at http://127.4.58.65:33965/ using document root <none> and password file <none>
I20260620 14:34:27.571404 21399 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:27.571471 21399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:27.571669 21399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:27.573760 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "ccf8f1f05f05478a86a0ffc4fd658912"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "ba354986bdc2458dbafe59f82c98645c"
server_key_iv: "ade8537184ae96723cb40e88554d7d8c"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.574311 21399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf8f1f05f05478a86a0ffc4fd658912"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "ba354986bdc2458dbafe59f82c98645c"
server_key_iv: "ade8537184ae96723cb40e88554d7d8c"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.578112 21399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:34:27.580585 21414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.581866 21399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:34:27.581976 21399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "ccf8f1f05f05478a86a0ffc4fd658912"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "ba354986bdc2458dbafe59f82c98645c"
server_key_iv: "ade8537184ae96723cb40e88554d7d8c"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.582079 21399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:27.606509 21399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:27.609519 21399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:27.609695 21399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:27.610288 21399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:27.611104 21399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:27.611151 21399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.611205 21399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:27.611231 21399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.620044 21399 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:41655
I20260620 14:34:27.620072 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:41655 every 8 connection(s)
I20260620 14:34:27.621028 21399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:27.628033 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21399
I20260620 14:34:27.628160 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:27.628479 4329 external_mini_cluster.cc:1428] Setting key 901f63ac97e86fa790d473d206b24e76
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:27.634899 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:27.622847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:58437 (local address 127.4.58.126:33855)
0620 14:34:27.622975 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:27.622978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:27.623847 (+ 869us) server_negotiation.cc:407] Connection header received
0620 14:34:27.625175 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:27.625179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:27.625243 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:27.625323 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:27.626561 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.627099 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.628307 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.628547 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.628641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:27.631275 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:27.631279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:27.631281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:27.631310 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:27.632663 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.633263 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.633265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.633267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.633345 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.633688 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.633690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.633691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.633922 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:27.634012 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:27.634641 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 14:34:27.634749 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:34:27.636135 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:27.623227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33855 (local address 127.4.58.65:58437)
0620 14:34:27.623681 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:27.623726 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:27.624949 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:27.625527 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:27.625534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:27.625901 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:27.626404 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:27.626418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.627223 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.627226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:27.628184 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:27.628196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.628736 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.628741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:27.628976 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:27.629529 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:27.629549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:27.631105 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:27.632804 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:27.632809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:27.632827 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:27.633167 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:27.633419 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:27.633422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:27.633424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:27.633602 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:27.634040 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:27.634047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:27.634395 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 14:34:27.634611 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 14:34:27.635032 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260620 14:34:27.637621 21528 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33855
I20260620 14:34:27.637884 21528 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:34:27.638392 21528 heartbeater.cc:511] Master 127.4.58.126:33855 requested a full tablet report, sending...
I20260620 14:34:27.639930 21337 ts_manager.cc:194] Registered new tserver with Master: ccf8f1f05f05478a86a0ffc4fd658912 (127.4.58.65:41655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
I20260620 14:34:27.641121 21337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:58437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:34:27.655673 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:27.770666 21533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:27.771023 21533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:27.771114 21533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:27.775452 21533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:27.775527 21533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:27.775614 21533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:27.781307 21533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:27.782438 21533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:27.783511 21533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:27.793504 21538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:27.793483 21539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:27.793678 21533 server_base.cc:1034] running on GCE node
W20260620 14:34:27.793483 21541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:27.794137 21533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:27.794768 21533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:27.795944 21533 hybrid_clock.cc:648] HybridClock initialized: now 1781966067795925 us; error 31 us; skew 500 ppm
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:27.798620 21533 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:34:27.799485 21533 webserver.cc:460] Webserver started at http://127.4.58.66:42277/ using document root <none> and password file <none>
I20260620 14:34:27.800015 21533 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:27.800079 21533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:27.800261 21533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:27.802302 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "e5ae82a1657b4f7aabd1fa028822f95c"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "159c64e52ec315505d25f9c63472b43b"
server_key_iv: "eec90bf4ab8e51e0ab16623999a13f95"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.802825 21533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "e5ae82a1657b4f7aabd1fa028822f95c"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "159c64e52ec315505d25f9c63472b43b"
server_key_iv: "eec90bf4ab8e51e0ab16623999a13f95"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.806473 21533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:34:27.808823 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.810189 21533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:27.810303 21533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "e5ae82a1657b4f7aabd1fa028822f95c"
format_stamp: "Formatted at 2026-06-20 14:34:27 on dist-test-slave-gqzk"
server_key: "159c64e52ec315505d25f9c63472b43b"
server_key_iv: "eec90bf4ab8e51e0ab16623999a13f95"
server_key_version: "encryptionkey@0"
I20260620 14:34:27.810425 21533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:27.834563 21533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:27.837783 21533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:27.837980 21533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:27.838567 21533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:27.839409 21533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:27.839457 21533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.839504 21533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:27.839531 21533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:27.848269 21533 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:44747
I20260620 14:34:27.848299 21661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:44747 every 8 connection(s)
I20260620 14:34:27.849247 21533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:27.851963 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21533
I20260620 14:34:27.852119 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:27.852444 4329 external_mini_cluster.cc:1428] Setting key 3fb64ecf04e93f7a770fd3ec1e589e11
Jun 20 14:34:27 dist-test-slave-gqzk krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966067, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:34:27.865047 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:27.851093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:52943 (local address 127.4.58.126:33855)
0620 14:34:27.851223 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:27.851226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:27.852213 (+ 987us) server_negotiation.cc:407] Connection header received
0620 14:34:27.853703 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:27.853707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:27.853756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:27.853870 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:27.855509 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.856046 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.857795 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.858036 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.858130 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:27.861185 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:27.861188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:27.861190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:27.861222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:27.862721 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.863241 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.863244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.863245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.863357 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:27.863754 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:27.863758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:27.863759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:27.863960 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:27.864047 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:27.864664 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 14:34:27.864877 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:34:27.865875 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:27.851508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33855 (local address 127.4.58.66:52943)
0620 14:34:27.852028 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:27.852093 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:27.853484 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:27.854094 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:27.854104 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:27.854623 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:27.855338 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:27.855358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.856178 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.856182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:27.857625 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:27.857638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:27.858219 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:27.858224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:27.858451 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:27.859113 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:27.859137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:27.860988 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:27.862837 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:27.862844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:27.862862 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:27.863129 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:27.863442 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:27.863446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:27.863448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:27.863659 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:27.864063 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:27.864070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:27.864399 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 14:34:27.864592 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:34:27.864977 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":132,"threads_started":1}
I20260620 14:34:27.867357 21662 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33855
I20260620 14:34:27.867606 21662 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:27.868114 21662 heartbeater.cc:511] Master 127.4.58.126:33855 requested a full tablet report, sending...
I20260620 14:34:27.869223 21337 ts_manager.cc:194] Registered new tserver with Master: e5ae82a1657b4f7aabd1fa028822f95c (127.4.58.66:44747)
I20260620 14:34:27.869828 21337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:52943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:34:27.880028 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:27.990325 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:27.990571 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:27.990629 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:27.994594 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:27.994653 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:27.994728 21667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:27.999933 21667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:28.000975 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:28.001977 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:28.011704 21667 server_base.cc:1034] running on GCE node
W20260620 14:34:28.011814 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:28.011718 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:28.011688 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:28.012351 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:28.013022 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:28.014176 21667 hybrid_clock.cc:648] HybridClock initialized: now 1781966068014143 us; error 44 us; skew 500 ppm
Jun 20 14:34:28 dist-test-slave-gqzk krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966068, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:28.016860 21667 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:34:28.017777 21667 webserver.cc:460] Webserver started at http://127.4.58.67:39545/ using document root <none> and password file <none>
I20260620 14:34:28.018354 21667 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:28.018417 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:28.018616 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:28.020674 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "d4b4168bdb764eed8d3df4aa938ef78c"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "d655c2d90cf49f3984cba5d1d693d880"
server_key_iv: "160820dde75875f84b22cd427dba65ee"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.021267 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b4168bdb764eed8d3df4aa938ef78c"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "d655c2d90cf49f3984cba5d1d693d880"
server_key_iv: "160820dde75875f84b22cd427dba65ee"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.024961 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:28.027350 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.028548 21667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:28.028659 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "d4b4168bdb764eed8d3df4aa938ef78c"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "d655c2d90cf49f3984cba5d1d693d880"
server_key_iv: "160820dde75875f84b22cd427dba65ee"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.028808 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:28.048586 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:28.051728 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:28.051910 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:28.052507 21667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:28.053382 21667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:28.053431 21667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.053469 21667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:28.053496 21667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.061945 21667 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:40337
I20260620 14:34:28.061977 21795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:40337 every 8 connection(s)
I20260620 14:34:28.062831 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:28.066144 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21667
I20260620 14:34:28.066255 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:28.066519 4329 external_mini_cluster.cc:1428] Setting key fc7fe8f326deb513aee18ffbfcb9f2aa
Jun 20 14:34:28 dist-test-slave-gqzk krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966068, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:28.075968 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.064610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:36521 (local address 127.4.58.126:33855)
0620 14:34:28.064778 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.064783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.065645 (+ 862us) server_negotiation.cc:407] Connection header received
0620 14:34:28.066980 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.066984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.067040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.067116 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:28.068282 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.068864 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.069983 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.070202 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.070289 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.072599 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:28.072603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:28.072605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:28.072631 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:28.074011 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:28.074499 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:28.074503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:28.074505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:28.074580 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:28.074889 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:28.074893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:28.074895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:28.075078 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:28.075162 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.075670 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.075800 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:34:28.076820 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.065050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33855 (local address 127.4.58.67:36521)
0620 14:34:28.065495 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.065537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.066768 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.067306 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.067313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.067666 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:28.068142 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.068155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.068976 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.068979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.069864 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.069873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.070344 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.070348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.070505 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.070992 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:28.071008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:28.072427 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:28.074119 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:28.074126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:28.074142 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:28.074408 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:28.074660 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:28.074663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:28.074665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:28.074805 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:28.075173 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:28.075186 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:28.075443 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.075613 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.075943 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260620 14:34:28.078043 21796 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33855
I20260620 14:34:28.078317 21796 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:28.078852 21796 heartbeater.cc:511] Master 127.4.58.126:33855 requested a full tablet report, sending...
I20260620 14:34:28.079944 21336 ts_manager.cc:194] Registered new tserver with Master: d4b4168bdb764eed8d3df4aa938ef78c (127.4.58.67:40337)
I20260620 14:34:28.080616 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:36521
I20260620 14:34:28.091958 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:28.101106 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.094099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.4.58.126:33855)
0620 14:34:28.094310 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.094314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.094405 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:34:28.094513 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.094516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.094556 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.094618 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:28.095247 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.095755 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.096572 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.096809 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.096888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.098136 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:28.098139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:28.098141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:28.098163 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:28.099437 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:28.099855 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:28.099858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:28.099859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:28.099929 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:28.100218 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:28.100220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:28.100221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:28.100465 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:28.100568 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.100880 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.100982 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:34:28.104132 21336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:28.106995 21336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:28.120661 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.115281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41655 (local address 127.0.0.1:57094)
0620 14:34:28.115496 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.115508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.115638 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.116790 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.116799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.116841 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:28.117025 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.117031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.118061 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.118064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.119223 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.119232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.120110 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.120114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.120229 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.120273 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.120391 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.120507 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":64,"spinlock_wait_cycles":2688,"thread_start_us":90,"threads_started":1}
I20260620 14:34:28.120682 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.115761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.4.58.65:41655)
0620 14:34:28.116283 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.116298 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.116322 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:34:28.116400 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.116406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.116611 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.116972 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:28.117158 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.117936 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.119342 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.119994 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.120096 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.120237 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.120429 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.120524 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":125,"threads_started":1}
I20260620 14:34:28.121048 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.115686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:44747 (local address 127.0.0.1:43642)
0620 14:34:28.115923 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.115935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.116011 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.116798 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.116801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.116837 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:28.116983 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.116991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.118439 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.118443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.119624 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.119634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.120621 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.120625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.120709 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.120735 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.120823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.120914 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"spinlock_wait_cycles":4352,"thread_start_us":103,"threads_started":1}
I20260620 14:34:28.121291 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.114960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:40337 (local address 127.0.0.1:33016)
0620 14:34:28.115321 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.115374 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.116161 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.117080 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.117083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.117100 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:28.117181 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.117187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.118582 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.118585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.119672 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.119679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.120723 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.120727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.120844 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.120867 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.120933 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.121001 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260620 14:34:28.121342 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.115057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.4.58.67:40337)
0620 14:34:28.116487 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.116497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.116516 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:34:28.116580 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.116585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.116738 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.116910 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:28.117298 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.118452 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.119795 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.120457 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.120540 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.120664 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.121109 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.121206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":159,"threads_started":1}
I20260620 14:34:28.121865 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.116066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.4.58.66:44747)
0620 14:34:28.116390 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.116400 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.116418 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:34:28.116477 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.116481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.116621 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.116773 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:28.117132 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.118301 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.119747 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.120493 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.121440 (+ 947us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.121574 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.121622 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.121705 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":110,"threads_started":1}
I20260620 14:34:28.124017 21462 tablet_service.cc:1467] Processing CreateTablet for tablet 273b6b9d2da54affb5b72bb40b961981 (DEFAULT_TABLE table=test-table [id=ecf3ede524cb4d439c6ed10dfdd695cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:28.124017 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 273b6b9d2da54affb5b72bb40b961981 (DEFAULT_TABLE table=test-table [id=ecf3ede524cb4d439c6ed10dfdd695cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:28.124240 21596 tablet_service.cc:1467] Processing CreateTablet for tablet 273b6b9d2da54affb5b72bb40b961981 (DEFAULT_TABLE table=test-table [id=ecf3ede524cb4d439c6ed10dfdd695cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:28.125144 21596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273b6b9d2da54affb5b72bb40b961981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:28.125162 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273b6b9d2da54affb5b72bb40b961981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:28.125162 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273b6b9d2da54affb5b72bb40b961981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:28.132393 21814 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Bootstrap starting.
I20260620 14:34:28.132580 21815 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Bootstrap starting.
I20260620 14:34:28.133060 21813 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Bootstrap starting.
I20260620 14:34:28.134557 21814 tablet_bootstrap.cc:654] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:28.134821 21815 tablet_bootstrap.cc:654] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:28.135505 21813 tablet_bootstrap.cc:654] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:28.135816 21814 log.cc:824] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:28.136114 21815 log.cc:824] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:28.136267 21813 log.cc:824] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:28.137995 21814 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: No bootstrap required, opened a new log
I20260620 14:34:28.138200 21814 ts_tablet_manager.cc:1397] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 14:34:28.138839 21813 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: No bootstrap required, opened a new log
I20260620 14:34:28.138950 21815 tablet_bootstrap.cc:492] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: No bootstrap required, opened a new log
I20260620 14:34:28.139034 21813 ts_tablet_manager.cc:1397] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:34:28.139144 21815 ts_tablet_manager.cc:1397] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:34:28.142040 21813 raft_consensus.cc:348] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.142306 21813 raft_consensus.cc:374] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:28.142133 21814 raft_consensus.cc:348] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.142369 21813 raft_consensus.cc:729] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ae82a1657b4f7aabd1fa028822f95c, State: Initialized, Role: FOLLOWER
I20260620 14:34:28.142395 21814 raft_consensus.cc:374] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:28.142442 21814 raft_consensus.cc:729] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b4168bdb764eed8d3df4aa938ef78c, State: Initialized, Role: FOLLOWER
I20260620 14:34:28.142835 21813 consensus_queue.cc:260] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.142763 21815 raft_consensus.cc:348] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.142894 21814 consensus_queue.cc:260] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.142980 21815 raft_consensus.cc:374] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:28.143020 21815 raft_consensus.cc:729] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf8f1f05f05478a86a0ffc4fd658912, State: Initialized, Role: FOLLOWER
I20260620 14:34:28.143435 21815 consensus_queue.cc:260] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.143589 21796 heartbeater.cc:503] Master 127.4.58.126:33855 was elected leader, sending a full tablet report...
I20260620 14:34:28.143893 21814 ts_tablet_manager.cc:1428] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 14:34:28.144240 21815 ts_tablet_manager.cc:1428] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:34:28.144276 21813 ts_tablet_manager.cc:1428] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:34:28.144557 21528 heartbeater.cc:503] Master 127.4.58.126:33855 was elected leader, sending a full tablet report...
I20260620 14:34:28.144692 21662 heartbeater.cc:503] Master 127.4.58.126:33855 was elected leader, sending a full tablet report...
W20260620 14:34:28.313829 21797 tablet.cc:2369] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:34:28.350359 21663 tablet.cc:2369] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:34:28.372543 21529 tablet.cc:2369] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:28.422747 21820 raft_consensus.cc:482] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:28.422955 21820 raft_consensus.cc:504] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.424118 21820 leader_election.cc:288] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ae82a1657b4f7aabd1fa028822f95c (127.4.58.66:44747), ccf8f1f05f05478a86a0ffc4fd658912 (127.4.58.65:41655)
I20260620 14:34:28.427678 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.424609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:56181 (local address 127.4.58.65:41655)
0620 14:34:28.424743 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.424764 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.424775 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:34:28.424840 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.424842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.424879 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.424946 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:28.425222 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.425715 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.426760 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.427148 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.427238 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.427302 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.427486 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.427558 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:34:28.427678 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.424389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41655 (local address 127.4.58.67:56181)
0620 14:34:28.424503 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.424520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.424615 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.424993 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.424996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.425013 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:28.425117 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.425122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.425824 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.425827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.426641 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.426650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.427264 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.427267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.427378 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.427395 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.427458 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.427541 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":22}
I20260620 14:34:28.427796 21812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.424434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:57127 (local address 127.4.58.66:44747)
0620 14:34:28.424557 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.424562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.424942 (+ 380us) server_negotiation.cc:407] Connection header received
0620 14:34:28.425050 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.425053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.425094 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.425154 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:28.425414 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.425889 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.426915 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.427294 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.427380 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.427438 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.427609 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.427678 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:34:28.427853 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.424389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:44747 (local address 127.4.58.67:57127)
0620 14:34:28.424855 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.424867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.424966 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.425175 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.425178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.425192 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:28.425303 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.425308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.425997 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.426001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.426805 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.426813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.427402 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.427405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.427508 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.427526 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.427588 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.427653 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":121,"thread_start_us":114,"threads_started":1}
I20260620 14:34:28.428368 21482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273b6b9d2da54affb5b72bb40b961981" candidate_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" is_pre_election: true
I20260620 14:34:28.428434 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273b6b9d2da54affb5b72bb40b961981" candidate_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" is_pre_election: true
I20260620 14:34:28.428665 21482 raft_consensus.cc:2454] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4b4168bdb764eed8d3df4aa938ef78c in term 0.
I20260620 14:34:28.428690 21616 raft_consensus.cc:2454] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4b4168bdb764eed8d3df4aa938ef78c in term 0.
I20260620 14:34:28.429178 21685 leader_election.cc:302] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: ccf8f1f05f05478a86a0ffc4fd658912, d4b4168bdb764eed8d3df4aa938ef78c, e5ae82a1657b4f7aabd1fa028822f95c; no voters:
I20260620 14:34:28.429433 21820 raft_consensus.cc:2781] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:28.429513 21820 raft_consensus.cc:482] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:28.429561 21820 raft_consensus.cc:3037] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:28.430521 21820 raft_consensus.cc:504] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.430840 21820 leader_election.cc:288] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [CANDIDATE]: Term 1 election: Requested vote from peers e5ae82a1657b4f7aabd1fa028822f95c (127.4.58.66:44747), ccf8f1f05f05478a86a0ffc4fd658912 (127.4.58.65:41655)
I20260620 14:34:28.431175 21616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273b6b9d2da54affb5b72bb40b961981" candidate_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ae82a1657b4f7aabd1fa028822f95c"
I20260620 14:34:28.431216 21482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273b6b9d2da54affb5b72bb40b961981" candidate_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccf8f1f05f05478a86a0ffc4fd658912"
I20260620 14:34:28.431298 21616 raft_consensus.cc:3037] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:28.431324 21482 raft_consensus.cc:3037] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:28.432307 21482 raft_consensus.cc:2454] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b4168bdb764eed8d3df4aa938ef78c in term 1.
I20260620 14:34:28.432395 21616 raft_consensus.cc:2454] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b4168bdb764eed8d3df4aa938ef78c in term 1.
I20260620 14:34:28.432602 21684 leader_election.cc:302] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf8f1f05f05478a86a0ffc4fd658912, d4b4168bdb764eed8d3df4aa938ef78c; no voters:
I20260620 14:34:28.432798 21820 raft_consensus.cc:2781] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:28.433014 21820 raft_consensus.cc:686] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 1 LEADER]: Becoming Leader. State: Replica: d4b4168bdb764eed8d3df4aa938ef78c, State: Running, Role: LEADER
I20260620 14:34:28.433363 21820 consensus_queue.cc:237] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } }
I20260620 14:34:28.436477 21336 catalog_manager.cc:5495] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c reported cstate change: term changed from 0 to 1, leader changed from <none> to d4b4168bdb764eed8d3df4aa938ef78c (127.4.58.67). New cstate: current_term: 1 leader_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b4168bdb764eed8d3df4aa938ef78c" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 } health_report { overall_health: UNKNOWN } } }
I20260620 14:34:28.484571 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.481005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.4.58.67:40337)
0620 14:34:28.481147 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.481151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.481274 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:34:28.481418 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.481421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.481481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.481572 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:28.481898 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.482388 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.483224 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.483425 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.483493 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.483758 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:28.484189 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:28.484275 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.484359 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.484411 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:34:28.490514 21482 raft_consensus.cc:1261] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 1 FOLLOWER]: Refusing update from remote peer d4b4168bdb764eed8d3df4aa938ef78c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:28.490514 21616 raft_consensus.cc:1261] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 1 FOLLOWER]: Refusing update from remote peer d4b4168bdb764eed8d3df4aa938ef78c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:28.491163 21823 consensus_queue.cc:1035] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccf8f1f05f05478a86a0ffc4fd658912" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 41655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:28.491374 21820 consensus_queue.cc:1035] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ae82a1657b4f7aabd1fa028822f95c" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:28.501966 21827 mvcc.cc:205] Tried to move back new op lower bound from 7298933016530071552 to 7298933016305369088. Current Snapshot: MvccSnapshot[applied={T|T < 7298933016530071552 or (T in {7298933016530071552})}]
I20260620 14:34:28.507436 21336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:28.507594 21336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:28.509763 21336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be1908d5e05343e2a5978e4af9fbbe56: Sending DeleteTablet for 3 replicas of tablet 273b6b9d2da54affb5b72bb40b961981
I20260620 14:34:28.510505 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 273b6b9d2da54affb5b72bb40b961981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:28 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33016
I20260620 14:34:28.510537 21596 tablet_service.cc:1514] Processing DeleteTablet for tablet 273b6b9d2da54affb5b72bb40b961981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:28 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:43642
I20260620 14:34:28.510638 21462 tablet_service.cc:1514] Processing DeleteTablet for tablet 273b6b9d2da54affb5b72bb40b961981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:28 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:57094
I20260620 14:34:28.511020 21836 tablet_replica.cc:330] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: stopping tablet replica
I20260620 14:34:28.511322 21837 tablet_replica.cc:330] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912: stopping tablet replica
I20260620 14:34:28.511343 21836 raft_consensus.cc:2229] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:28.511427 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21399
I20260620 14:34:28.511605 21837 raft_consensus.cc:2229] T 273b6b9d2da54affb5b72bb40b961981 P ccf8f1f05f05478a86a0ffc4fd658912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:28.511646 21836 raft_consensus.cc:2258] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:28.512355 21835 tablet_replica.cc:330] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: stopping tablet replica
I20260620 14:34:28.512619 21835 raft_consensus.cc:2229] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:28.512979 21835 raft_consensus.cc:2258] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:28.514569 21836 ts_tablet_manager.cc:1905] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:28.514849 21835 ts_tablet_manager.cc:1905] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:28.517452 21836 ts_tablet_manager.cc:1918] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:28.517495 21835 ts_tablet_manager.cc:1918] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:28.517540 21836 log.cc:1196] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/273b6b9d2da54affb5b72bb40b961981
I20260620 14:34:28.517581 21835 log.cc:1196] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/273b6b9d2da54affb5b72bb40b961981
I20260620 14:34:28.517848 21836 ts_tablet_manager.cc:1939] T 273b6b9d2da54affb5b72bb40b961981 P e5ae82a1657b4f7aabd1fa028822f95c: Deleting consensus metadata
I20260620 14:34:28.517940 21835 ts_tablet_manager.cc:1939] T 273b6b9d2da54affb5b72bb40b961981 P d4b4168bdb764eed8d3df4aa938ef78c: Deleting consensus metadata
I20260620 14:34:28.519104 21323 catalog_manager.cc:4841] TS d4b4168bdb764eed8d3df4aa938ef78c (127.4.58.67:40337): tablet 273b6b9d2da54affb5b72bb40b961981 (table test-table [id=ecf3ede524cb4d439c6ed10dfdd695cf]) successfully deleted
I20260620 14:34:28.519315 21323 catalog_manager.cc:4841] TS e5ae82a1657b4f7aabd1fa028822f95c (127.4.58.66:44747): tablet 273b6b9d2da54affb5b72bb40b961981 (table test-table [id=ecf3ede524cb4d439c6ed10dfdd695cf]) successfully deleted
W20260620 14:34:28.519516 21322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:41655 (error 108)
I20260620 14:34:28.520218 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21533
I20260620 14:34:28.520582 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.520282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:41655 (local address 127.0.0.1:57100)
0620 14:34:28.520416 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.520491 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:41655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:34:28.521914 21322 catalog_manager.cc:4584] TS ccf8f1f05f05478a86a0ffc4fd658912 (127.4.58.65:41655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 273b6b9d2da54affb5b72bb40b961981: Network error: Client connection negotiation failed: client connection to 127.4.58.65:41655: connect: Connection refused (error 111)
I20260620 14:34:28.525381 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21667
I20260620 14:34:28.532478 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21305
2026-06-20T14:34:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2453 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:28Z Disabled control of system clock
I20260620 14:34:28.560680 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40989
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:28.668167 21842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:28.668464 21842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:28.668500 21842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:28.672552 21842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:28.672634 21842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:28.672649 21842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:28.672664 21842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:28.672678 21842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:28.677592 21842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:40989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:28.678718 21842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:28.680090 21842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:28.688472 21848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:28.688452 21847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:28.688601 21842 server_base.cc:1034] running on GCE node
W20260620 14:34:28.688452 21850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:28.689327 21842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:28.690461 21842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:28.691609 21842 hybrid_clock.cc:648] HybridClock initialized: now 1781966068691579 us; error 45 us; skew 500 ppm
I20260620 14:34:28.693441 21842 webserver.cc:460] Webserver started at http://127.4.58.126:40133/ using document root <none> and password file <none>
I20260620 14:34:28.693992 21842 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:28.694053 21842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:28.694254 21842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:28.696277 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "fbfeda2511204684afa83a6b9382a15f"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "f1adfd8d2be7672a242e50d84d60056a"
server_key_iv: "3f3b61f8c733a90b08a0293f6ece22f3"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.696853 21842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "fbfeda2511204684afa83a6b9382a15f"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "f1adfd8d2be7672a242e50d84d60056a"
server_key_iv: "3f3b61f8c733a90b08a0293f6ece22f3"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.700344 21842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:34:28.702704 21856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.703935 21842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:28.704046 21842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "fbfeda2511204684afa83a6b9382a15f"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "f1adfd8d2be7672a242e50d84d60056a"
server_key_iv: "3f3b61f8c733a90b08a0293f6ece22f3"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.704138 21842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:28.714628 21842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:28.715265 21842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:28.715435 21842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:28.722074 21842 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:40989
I20260620 14:34:28.722091 21908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:40989 every 8 connection(s)
I20260620 14:34:28.723013 21842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:28.725876 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21842
I20260620 14:34:28.726008 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:28.726303 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:28.726337 4329 external_mini_cluster.cc:1428] Setting key db87d7a701cd4d000e047af2674a2f40
I20260620 14:34:28.732530 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: Bootstrap starting.
I20260620 14:34:28.734709 21909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:28.735070 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.727729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.4.58.126:40989)
0620 14:34:28.728361 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.728376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.728409 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:34:28.729058 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.729074 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.729464 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.729765 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:28.730186 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.731127 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.731974 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.732442 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.732585 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.733411 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:28.733416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:28.733428 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:28.733801 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:28.733856 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:28.733875 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:28.733960 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.734091 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.734346 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260620 14:34:28.735497 21909 log.cc:824] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:28.737488 21909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: No bootstrap required, opened a new log
I20260620 14:34:28.740998 21909 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } }
I20260620 14:34:28.741221 21909 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:28.741276 21909 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfeda2511204684afa83a6b9382a15f, State: Initialized, Role: FOLLOWER
I20260620 14:34:28.741670 21909 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } }
I20260620 14:34:28.741793 21909 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:28.741858 21909 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:28.741952 21909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:28.742805 21909 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } }
I20260620 14:34:28.743144 21909 leader_election.cc:302] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbfeda2511204684afa83a6b9382a15f; no voters:
I20260620 14:34:28.743469 21909 leader_election.cc:288] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:28.743580 21914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:28.743789 21914 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [term 1 LEADER]: Becoming Leader. State: Replica: fbfeda2511204684afa83a6b9382a15f, State: Running, Role: LEADER
I20260620 14:34:28.744086 21914 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } }
I20260620 14:34:28.744673 21909 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:28.745859 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbfeda2511204684afa83a6b9382a15f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } } }
I20260620 14:34:28.745864 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbfeda2511204684afa83a6b9382a15f. Latest consensus state: current_term: 1 leader_uuid: "fbfeda2511204684afa83a6b9382a15f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbfeda2511204684afa83a6b9382a15f" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40989 } } }
I20260620 14:34:28.745995 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:28.745994 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:28.746428 21923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:28.749090 21923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:28.754207 21923 catalog_manager.cc:1347] Generated new cluster ID: 19601291b50e478194cf1b0cb485e5cd
I20260620 14:34:28.754282 21923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:28.771945 21923 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:28.773360 21923 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:28.783984 21923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: Generated new TSK 0
I20260620 14:34:28.784929 21923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:28.799984 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--builtin_ntp_servers=127.4.58.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:28.909919 21933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:28.910189 21933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:28.910233 21933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:28.914340 21933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:28.914403 21933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:28.914467 21933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:28.919318 21933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:28.920429 21933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:28.921810 21933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:28.931144 21939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:28.931193 21933 server_base.cc:1034] running on GCE node
W20260620 14:34:28.931141 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:28.931135 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:28.931806 21933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:28.932478 21933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:28.933657 21933 hybrid_clock.cc:648] HybridClock initialized: now 1781966068933612 us; error 56 us; skew 500 ppm
I20260620 14:34:28.935491 21933 webserver.cc:460] Webserver started at http://127.4.58.65:43851/ using document root <none> and password file <none>
I20260620 14:34:28.936076 21933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:28.936146 21933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:28.936352 21933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:28.938407 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "5c9c526d772548c0bb8b3e73e6a13fb3"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "ef1b2e5bff4c82efd9bb9b3a492275ca"
server_key_iv: "890d6c7311b83713c07c331cb9871f60"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.938920 21933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "5c9c526d772548c0bb8b3e73e6a13fb3"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "ef1b2e5bff4c82efd9bb9b3a492275ca"
server_key_iv: "890d6c7311b83713c07c331cb9871f60"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.942507 21933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:34:28.944823 21947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.946111 21933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:28.946223 21933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "5c9c526d772548c0bb8b3e73e6a13fb3"
format_stamp: "Formatted at 2026-06-20 14:34:28 on dist-test-slave-gqzk"
server_key: "ef1b2e5bff4c82efd9bb9b3a492275ca"
server_key_iv: "890d6c7311b83713c07c331cb9871f60"
server_key_version: "encryptionkey@0"
I20260620 14:34:28.946329 21933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:28.966264 21933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:28.966956 21933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:28.967124 21933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:28.967691 21933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:28.968611 21933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:28.968657 21933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.968708 21933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:28.968734 21933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:28.976987 21933 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:37769
I20260620 14:34:28.977013 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:37769 every 8 connection(s)
I20260620 14:34:28.977891 21933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:28.986232 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 21933
I20260620 14:34:28.986387 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:28.986689 4329 external_mini_cluster.cc:1428] Setting key c5310471d566a8c5f391b11063085fe0
I20260620 14:34:28.986972 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.979654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:47707 (local address 127.4.58.126:40989)
0620 14:34:28.979795 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:28.979799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:28.980862 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 14:34:28.982286 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:28.982289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:28.982346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:28.982419 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:28.983450 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.983981 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.985149 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.985370 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.985441 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.986506 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:28.986510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:28.986512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:28.986525 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:28.986545 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:28.986553 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:28.986608 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:28.986768 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:34:28.986854 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:34:28.987973 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:28.980055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40989 (local address 127.4.58.65:47707)
0620 14:34:28.980696 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:28.980736 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:28.982078 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:28.982597 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:28.982604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:28.982691 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:28.983294 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.983313 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.984094 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.984097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.985043 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:28.985052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:28.985487 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:28.985489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:28.985628 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:28.986201 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:28.986203 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:28.986259 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:28.986345 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:28.986626 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:28.986635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:28.986641 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:34:28.986768 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:34:28.987072 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260620 14:34:28.988627 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--builtin_ntp_servers=127.4.58.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:28.989167 22061 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40989
I20260620 14:34:28.989406 22061 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:28.989913 22061 heartbeater.cc:511] Master 127.4.58.126:40989 requested a full tablet report, sending...
I20260620 14:34:28.991353 21873 ts_manager.cc:194] Registered new tserver with Master: 5c9c526d772548c0bb8b3e73e6a13fb3 (127.4.58.65:37769)
I20260620 14:34:28.992803 21873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.65:47707
W20260620 14:34:29.098531 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:29.098824 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:29.098866 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:29.102905 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:29.102982 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:29.103042 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:29.108000 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:29.109261 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:29.110807 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:29.120249 22072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:29.120385 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:29.120277 22070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:29.120409 22064 server_base.cc:1034] running on GCE node
I20260620 14:34:29.120828 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:29.121479 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:29.122643 22064 hybrid_clock.cc:648] HybridClock initialized: now 1781966069122617 us; error 36 us; skew 500 ppm
I20260620 14:34:29.124500 22064 webserver.cc:460] Webserver started at http://127.4.58.66:40789/ using document root <none> and password file <none>
I20260620 14:34:29.125137 22064 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:29.125213 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:29.125414 22064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:29.127485 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "b18b94a1112b4449aac0b046f748aa46"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "8f38801ab12e0ab6397e430a9689de03"
server_key_iv: "ead7903b8590dcdff754f07bc4ed6374"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.128036 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "b18b94a1112b4449aac0b046f748aa46"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "8f38801ab12e0ab6397e430a9689de03"
server_key_iv: "ead7903b8590dcdff754f07bc4ed6374"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.131762 22064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:34:29.134301 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.135515 22064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:29.135625 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "b18b94a1112b4449aac0b046f748aa46"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "8f38801ab12e0ab6397e430a9689de03"
server_key_iv: "ead7903b8590dcdff754f07bc4ed6374"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.135730 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:29.170349 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:29.171090 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:29.171267 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:29.171896 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:29.172905 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:29.172971 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.173084 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:29.173121 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.181557 22064 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:42375
I20260620 14:34:29.181581 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:42375 every 8 connection(s)
I20260620 14:34:29.182554 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:29.186394 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22064
I20260620 14:34:29.186534 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:29.186865 4329 external_mini_cluster.cc:1428] Setting key a512aa309b04209c13546920bca3f429
I20260620 14:34:29.188939 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--builtin_ntp_servers=127.4.58.84:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:29.192301 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.184397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:57325 (local address 127.4.58.126:40989)
0620 14:34:29.184565 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.184570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.185650 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 14:34:29.187250 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.187254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.187316 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.187408 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:29.188446 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.189001 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.190172 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.190516 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.190616 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.191688 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:29.191691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:29.191694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:29.191708 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:29.191732 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:29.191741 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:29.191812 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.192068 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.192138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:34:29.193475 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.184858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40989 (local address 127.4.58.66:57325)
0620 14:34:29.185501 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.185542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.187012 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.187617 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.187626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.187721 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:29.188303 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.188325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.189156 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.189158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.190054 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.190064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.190660 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.190663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.190824 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.191420 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:29.191421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:29.191465 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:29.191535 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:29.191895 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:29.191905 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:29.191911 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.192151 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.192477 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260620 14:34:29.194679 22192 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40989
I20260620 14:34:29.194950 22192 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:29.195636 22192 heartbeater.cc:511] Master 127.4.58.126:40989 requested a full tablet report, sending...
I20260620 14:34:29.196798 21873 ts_manager.cc:194] Registered new tserver with Master: b18b94a1112b4449aac0b046f748aa46 (127.4.58.66:42375)
I20260620 14:34:29.197365 21873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.66:57325
W20260620 14:34:29.300598 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:29.300943 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:29.300995 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:29.305135 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:29.305197 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:29.305261 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:29.310295 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:29.311446 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:29.312909 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:29.322678 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:29.322916 22195 server_base.cc:1034] running on GCE node
W20260620 14:34:29.322729 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:29.322712 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:29.323444 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:29.324082 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:29.325246 22195 hybrid_clock.cc:648] HybridClock initialized: now 1781966069325211 us; error 44 us; skew 500 ppm
I20260620 14:34:29.327057 22195 webserver.cc:460] Webserver started at http://127.4.58.67:38915/ using document root <none> and password file <none>
I20260620 14:34:29.327631 22195 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:29.327719 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:29.327937 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:29.330011 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "aa8baca4e35b4dee95cc642443948a15"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "9c0208c3540b41b8bf868f020af34efe"
server_key_iv: "70a0f01381c7c207667acd244b77e4cd"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.330542 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "aa8baca4e35b4dee95cc642443948a15"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "9c0208c3540b41b8bf868f020af34efe"
server_key_iv: "70a0f01381c7c207667acd244b77e4cd"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.334163 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:29.336687 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.337934 22195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:29.338042 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "aa8baca4e35b4dee95cc642443948a15"
format_stamp: "Formatted at 2026-06-20 14:34:29 on dist-test-slave-gqzk"
server_key: "9c0208c3540b41b8bf868f020af34efe"
server_key_iv: "70a0f01381c7c207667acd244b77e4cd"
server_key_version: "encryptionkey@0"
I20260620 14:34:29.338141 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:29.365016 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:29.365760 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:29.365937 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:29.366541 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:29.367509 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:29.367558 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.367609 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:29.367638 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:29.375887 22195 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:35371
I20260620 14:34:29.375916 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:35371 every 8 connection(s)
I20260620 14:34:29.376801 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:29.385571 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.378547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:33467 (local address 127.4.58.126:40989)
0620 14:34:29.378690 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.378695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.379803 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 14:34:29.381139 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.381142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.381189 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.381260 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:29.382086 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.382612 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.383763 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.383975 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.384063 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.385114 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:29.385118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:29.385120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:29.385131 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:29.385149 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:29.385157 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:29.385212 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.385362 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.385424 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:34:29.385742 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22195
I20260620 14:34:29.385860 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:29.386122 4329 external_mini_cluster.cc:1428] Setting key b62822e97e216b9295aca52820d964d4
I20260620 14:34:29.386471 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.379002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:40989 (local address 127.4.58.67:33467)
0620 14:34:29.379647 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.379688 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.380963 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.381422 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.381428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.381513 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:29.381981 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.381994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.382752 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.382756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.383636 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.383646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.384112 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.384116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.384270 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.384862 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:29.384864 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:29.384903 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:29.384965 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:29.385224 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:29.385233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:29.385239 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.385347 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.385623 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260620 14:34:29.387563 22323 heartbeater.cc:347] Connected to a master server at 127.4.58.126:40989
I20260620 14:34:29.387786 22323 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:29.388394 22323 heartbeater.cc:511] Master 127.4.58.126:40989 requested a full tablet report, sending...
I20260620 14:34:29.389546 21873 ts_manager.cc:194] Registered new tserver with Master: aa8baca4e35b4dee95cc642443948a15 (127.4.58.67:35371)
I20260620 14:34:29.390168 21873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.67:33467
I20260620 14:34:29.400490 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:29.406447 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.402614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.4.58.126:40989)
0620 14:34:29.402784 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.402788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.402934 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:34:29.403065 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.403068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.403113 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.403240 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:29.403532 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.404056 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.404972 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.405198 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.405287 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.406048 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:29.406051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:29.406053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:29.406066 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:29.406087 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:29.406095 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:29.406162 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.406241 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.406301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:34:29.409241 21873 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:29.412184 21873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:29.426373 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.420682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:42375 (local address 127.0.0.1:60560)
0620 14:34:29.421127 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.421178 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.421291 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.421860 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.421867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.421895 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:29.422040 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.422048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.423665 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.423668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.425013 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.425023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.425966 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.425969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.426046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.426077 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.426148 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.426241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":36,"thread_start_us":140,"threads_started":1}
I20260620 14:34:29.426483 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.420816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.4.58.66:42375)
0620 14:34:29.421308 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.421320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.421348 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:34:29.421434 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.421440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.421669 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.422204 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:29.422357 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.423561 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.425139 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.425860 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.425972 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.426166 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.426217 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.426318 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":149,"threads_started":1}
I20260620 14:34:29.427112 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.421807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:35371 (local address 127.0.0.1:57012)
0620 14:34:29.422400 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.422412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.422489 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.423145 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.423149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.423166 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:29.423331 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.423337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.424588 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.424591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.425793 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.425803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.426826 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.426829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.426895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.426915 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.426962 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.427013 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":63,"thread_start_us":84,"threads_started":1}
I20260620 14:34:29.429255 22126 tablet_service.cc:1467] Processing CreateTablet for tablet 3f64dc63628f47f3838a6b23819c88fd (DEFAULT_TABLE table=test-table [id=da6ee96bf7254fabae153ae3345b3ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:29.430202 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f64dc63628f47f3838a6b23819c88fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:29.432257 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.422210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.4.58.67:35371)
0620 14:34:29.422755 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.422767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.422812 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:34:29.422882 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.422886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.423029 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.423156 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:29.423438 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.424485 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.425913 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.426718 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.426828 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.426992 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.427035 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.427116 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":141,"threads_started":1}
I20260620 14:34:29.433708 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.421503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:37769 (local address 127.0.0.1:43392)
0620 14:34:29.421723 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.421736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.421848 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.423186 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.423189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.423206 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:29.423359 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.423365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.424481 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.424483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.425796 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.425803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.433287 (+ 7484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.433291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.433392 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.433419 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.433489 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.433568 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":52,"thread_start_us":91,"threads_started":1}
I20260620 14:34:29.434085 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.421643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.4.58.65:37769)
0620 14:34:29.422721 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.422731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.422764 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:34:29.422844 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.422849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.423039 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.423478 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:29.423606 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.424362 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.432143 (+ 7781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.433143 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.433264 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.433591 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.433645 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.433738 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":143,"threads_started":1}
I20260620 14:34:29.435246 22257 tablet_service.cc:1467] Processing CreateTablet for tablet 3f64dc63628f47f3838a6b23819c88fd (DEFAULT_TABLE table=test-table [id=da6ee96bf7254fabae153ae3345b3ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:29.436388 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f64dc63628f47f3838a6b23819c88fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:29.436723 22340 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: Bootstrap starting.
I20260620 14:34:29.437099 21995 tablet_service.cc:1467] Processing CreateTablet for tablet 3f64dc63628f47f3838a6b23819c88fd (DEFAULT_TABLE table=test-table [id=da6ee96bf7254fabae153ae3345b3ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:29.438169 21995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f64dc63628f47f3838a6b23819c88fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:29.438759 22340 tablet_bootstrap.cc:654] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:29.439496 22340 log.cc:824] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:29.441506 22340 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: No bootstrap required, opened a new log
I20260620 14:34:29.441697 22340 ts_tablet_manager.cc:1397] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 14:34:29.443768 22342 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Bootstrap starting.
I20260620 14:34:29.444962 22343 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Bootstrap starting.
I20260620 14:34:29.445794 22340 raft_consensus.cc:348] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.446077 22340 raft_consensus.cc:374] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:29.446180 22340 raft_consensus.cc:729] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18b94a1112b4449aac0b046f748aa46, State: Initialized, Role: FOLLOWER
I20260620 14:34:29.446511 22342 tablet_bootstrap.cc:654] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:29.446646 22340 consensus_queue.cc:260] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.446933 22343 tablet_bootstrap.cc:654] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:29.447368 22192 heartbeater.cc:503] Master 127.4.58.126:40989 was elected leader, sending a full tablet report...
I20260620 14:34:29.447728 22340 ts_tablet_manager.cc:1428] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:34:29.447758 22343 log.cc:824] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:29.447367 22342 log.cc:824] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:29.449940 22342 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: No bootstrap required, opened a new log
I20260620 14:34:29.450052 22343 tablet_bootstrap.cc:492] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: No bootstrap required, opened a new log
I20260620 14:34:29.450131 22342 ts_tablet_manager.cc:1397] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:34:29.450266 22343 ts_tablet_manager.cc:1397] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:34:29.453763 22342 raft_consensus.cc:348] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.454010 22342 raft_consensus.cc:374] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:29.453856 22343 raft_consensus.cc:348] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.454066 22342 raft_consensus.cc:729] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8baca4e35b4dee95cc642443948a15, State: Initialized, Role: FOLLOWER
I20260620 14:34:29.454107 22343 raft_consensus.cc:374] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:29.454164 22343 raft_consensus.cc:729] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9c526d772548c0bb8b3e73e6a13fb3, State: Initialized, Role: FOLLOWER
I20260620 14:34:29.454525 22342 consensus_queue.cc:260] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.454540 22343 consensus_queue.cc:260] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.455139 22323 heartbeater.cc:503] Master 127.4.58.126:40989 was elected leader, sending a full tablet report...
I20260620 14:34:29.455528 22342 ts_tablet_manager.cc:1428] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:34:29.455528 22343 ts_tablet_manager.cc:1428] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:34:29.455538 22061 heartbeater.cc:503] Master 127.4.58.126:40989 was elected leader, sending a full tablet report...
W20260620 14:34:29.479009 22062 tablet.cc:2369] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:29.576112 22347 raft_consensus.cc:482] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:29.576345 22347 raft_consensus.cc:504] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.577608 22347 leader_election.cc:288] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b18b94a1112b4449aac0b046f748aa46 (127.4.58.66:42375), 5c9c526d772548c0bb8b3e73e6a13fb3 (127.4.58.65:37769)
I20260620 14:34:29.581914 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.577811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:42375 (local address 127.4.58.67:53033)
0620 14:34:29.577955 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.577970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.578057 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.578386 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.578389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.578407 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:29.578525 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.578532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.579482 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.579484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.580685 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.580695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.581514 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.581517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.581605 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.581632 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.581703 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.581776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:34:29.581934 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.577934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:53033 (local address 127.4.58.66:42375)
0620 14:34:29.578051 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.578056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.578072 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:34:29.578145 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.578148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.578197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.578273 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:29.578631 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.579371 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.580833 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.581397 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.581500 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.581565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.581731 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.581810 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:34:29.581957 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.578085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:37769 (local address 127.4.58.67:46987)
0620 14:34:29.578402 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.578415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:29.578500 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:29.578722 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:29.578724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:29.578739 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:29.578837 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.578841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.579690 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.579693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.580837 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:29.580844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.581624 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.581627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:29.581708 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.581726 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:34:29.581804 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:34:29.581859 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260620 14:34:29.581986 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.578207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:46987 (local address 127.4.58.65:37769)
0620 14:34:29.578336 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.578340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.578480 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:34:29.578586 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.578589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.578639 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.578706 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:29.578930 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.579594 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.580928 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.581517 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.581607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.581651 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.581805 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.581869 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:34:29.582510 22015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f64dc63628f47f3838a6b23819c88fd" candidate_uuid: "aa8baca4e35b4dee95cc642443948a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" is_pre_election: true
I20260620 14:34:29.582551 22146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f64dc63628f47f3838a6b23819c88fd" candidate_uuid: "aa8baca4e35b4dee95cc642443948a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18b94a1112b4449aac0b046f748aa46" is_pre_election: true
I20260620 14:34:29.582809 22015 raft_consensus.cc:2454] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8baca4e35b4dee95cc642443948a15 in term 0.
I20260620 14:34:29.582809 22146 raft_consensus.cc:2454] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8baca4e35b4dee95cc642443948a15 in term 0.
I20260620 14:34:29.583323 22212 leader_election.cc:302] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9c526d772548c0bb8b3e73e6a13fb3, aa8baca4e35b4dee95cc642443948a15; no voters:
I20260620 14:34:29.583601 22347 raft_consensus.cc:2781] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:29.583694 22347 raft_consensus.cc:482] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:29.583765 22347 raft_consensus.cc:3037] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:29.584950 22347 raft_consensus.cc:504] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.585322 22347 leader_election.cc:288] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [CANDIDATE]: Term 1 election: Requested vote from peers b18b94a1112b4449aac0b046f748aa46 (127.4.58.66:42375), 5c9c526d772548c0bb8b3e73e6a13fb3 (127.4.58.65:37769)
I20260620 14:34:29.585757 22146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f64dc63628f47f3838a6b23819c88fd" candidate_uuid: "aa8baca4e35b4dee95cc642443948a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18b94a1112b4449aac0b046f748aa46"
I20260620 14:34:29.585816 22015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f64dc63628f47f3838a6b23819c88fd" candidate_uuid: "aa8baca4e35b4dee95cc642443948a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3"
I20260620 14:34:29.585896 22146 raft_consensus.cc:3037] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:29.585938 22015 raft_consensus.cc:3037] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:29.586989 22015 raft_consensus.cc:2454] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa8baca4e35b4dee95cc642443948a15 in term 1.
I20260620 14:34:29.587015 22146 raft_consensus.cc:2454] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa8baca4e35b4dee95cc642443948a15 in term 1.
I20260620 14:34:29.587321 22212 leader_election.cc:302] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8baca4e35b4dee95cc642443948a15, b18b94a1112b4449aac0b046f748aa46; no voters:
I20260620 14:34:29.587479 22347 raft_consensus.cc:2781] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:29.587684 22347 raft_consensus.cc:686] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 1 LEADER]: Becoming Leader. State: Replica: aa8baca4e35b4dee95cc642443948a15, State: Running, Role: LEADER
I20260620 14:34:29.588050 22347 consensus_queue.cc:237] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } }
I20260620 14:34:29.591331 21873 catalog_manager.cc:5495] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa8baca4e35b4dee95cc642443948a15 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "aa8baca4e35b4dee95cc642443948a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8baca4e35b4dee95cc642443948a15" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260620 14:34:29.621619 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.618227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.4.58.67:35371)
0620 14:34:29.618351 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:29.618355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:29.618367 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:34:29.618441 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:29.618443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:29.618495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:29.618579 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:29.618772 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.619262 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.620074 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:29.620321 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:29.620423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:29.620741 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:29.621255 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:29.621361 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:29.621436 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:34:29.621487 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:34:29.627571 22146 raft_consensus.cc:1261] T 3f64dc63628f47f3838a6b23819c88fd P b18b94a1112b4449aac0b046f748aa46 [term 1 FOLLOWER]: Refusing update from remote peer aa8baca4e35b4dee95cc642443948a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:29.627571 22015 raft_consensus.cc:1261] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3 [term 1 FOLLOWER]: Refusing update from remote peer aa8baca4e35b4dee95cc642443948a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 14:34:29.627646 22324 tablet.cc:2369] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:29.628314 22347 consensus_queue.cc:1035] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9c526d772548c0bb8b3e73e6a13fb3" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:29.628505 22350 consensus_queue.cc:1035] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18b94a1112b4449aac0b046f748aa46" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:29.636723 22354 mvcc.cc:205] Tried to move back new op lower bound from 7298933021186887680 to 7298933021035692032. Current Snapshot: MvccSnapshot[applied={T|T < 7298933021186887680 or (T in {7298933021186887680})}]
I20260620 14:34:29.636971 22355 mvcc.cc:205] Tried to move back new op lower bound from 7298933021186887680 to 7298933021035692032. Current Snapshot: MvccSnapshot[applied={T|T < 7298933021186887680}]
I20260620 14:34:29.643695 21872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:29.643850 21872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:29.646370 21872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbfeda2511204684afa83a6b9382a15f: Sending DeleteTablet for 3 replicas of tablet 3f64dc63628f47f3838a6b23819c88fd
I20260620 14:34:29.647217 22257 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f64dc63628f47f3838a6b23819c88fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:29 UTC) from {username='slave'} at 127.0.0.1:57012
I20260620 14:34:29.647287 21995 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f64dc63628f47f3838a6b23819c88fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:29 UTC) from {username='slave'} at 127.0.0.1:43392
I20260620 14:34:29.647733 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21933
I20260620 14:34:29.647809 22364 tablet_replica.cc:330] T 3f64dc63628f47f3838a6b23819c88fd P 5c9c526d772548c0bb8b3e73e6a13fb3: stopping tablet replica
I20260620 14:34:29.648294 22126 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f64dc63628f47f3838a6b23819c88fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:29 UTC) from {username='slave'} at 127.0.0.1:60560
I20260620 14:34:29.649435 22363 tablet_replica.cc:330] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: stopping tablet replica
I20260620 14:34:29.649758 22363 raft_consensus.cc:2229] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:29.650194 22363 raft_consensus.cc:2258] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:29.651613 22363 ts_tablet_manager.cc:1905] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:34:29.653383 21859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:37769 (error 108)
I20260620 14:34:29.653448 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22064
I20260620 14:34:29.655678 22363 ts_tablet_manager.cc:1918] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:29.655783 22363 log.cc:1196] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/3f64dc63628f47f3838a6b23819c88fd
I20260620 14:34:29.656096 22363 ts_tablet_manager.cc:1939] T 3f64dc63628f47f3838a6b23819c88fd P aa8baca4e35b4dee95cc642443948a15: Deleting consensus metadata
I20260620 14:34:29.657691 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.657347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:37769 (local address 127.0.0.1:43398)
0620 14:34:29.657531 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.657602 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:37769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 14:34:29.657891 21859 catalog_manager.cc:4584] TS 5c9c526d772548c0bb8b3e73e6a13fb3 (127.4.58.65:37769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f64dc63628f47f3838a6b23819c88fd: Network error: Client connection negotiation failed: client connection to 127.4.58.65:37769: connect: Connection refused (error 111)
I20260620 14:34:29.658293 21859 catalog_manager.cc:4841] TS aa8baca4e35b4dee95cc642443948a15 (127.4.58.67:35371): tablet 3f64dc63628f47f3838a6b23819c88fd (table test-table [id=da6ee96bf7254fabae153ae3345b3ffe]) successfully deleted
I20260620 14:34:29.659034 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22195
I20260620 14:34:29.660709 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:29.659199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:42375 (local address 127.0.0.1:60574)
0620 14:34:29.660589 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:29.660639 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:42375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1303}
I20260620 14:34:29.664573 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 21842
2026-06-20T14:34:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1131 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:29 dist-test-slave-gqzk krb5kdc[22369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:29 dist-test-slave-gqzk krb5kdc[22369](info): set up 2 sockets
Jun 20 14:34:29 dist-test-slave-gqzk krb5kdc[22369](info): commencing operation
krb5kdc: starting...
W20260620 14:34:30.616592 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:30 dist-test-slave-gqzk krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:30.736455 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34905
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:30.860879 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:30.861150 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:30.861192 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:30.865283 22382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:30.865351 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:30.865370 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:30.865386 22382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:30.865401 22382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:30.871083 22382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:30.872166 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:30.873219 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:30.881688 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:30.881630 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:30.881747 22382 server_base.cc:1034] running on GCE node
W20260620 14:34:30.881702 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:30.882421 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:30.883582 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:30.884743 22382 hybrid_clock.cc:648] HybridClock initialized: now 1781966070884723 us; error 39 us; skew 500 ppm
Jun 20 14:34:30 dist-test-slave-gqzk krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966070, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:30.887570 22382 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:30.888445 22382 webserver.cc:460] Webserver started at http://127.4.58.126:43951/ using document root <none> and password file <none>
I20260620 14:34:30.889040 22382 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:30.889242 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:30.889488 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:30.891577 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "20279e36eb7f4756abe53e51d7572458"
format_stamp: "Formatted at 2026-06-20 14:34:30 on dist-test-slave-gqzk"
server_key: "e109a5522700bccbe5281e87b61055c0"
server_key_iv: "80d18891bb3722f5ef5f2574c10370a9"
server_key_version: "encryptionkey@0"
I20260620 14:34:30.892125 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "20279e36eb7f4756abe53e51d7572458"
format_stamp: "Formatted at 2026-06-20 14:34:30 on dist-test-slave-gqzk"
server_key: "e109a5522700bccbe5281e87b61055c0"
server_key_iv: "80d18891bb3722f5ef5f2574c10370a9"
server_key_version: "encryptionkey@0"
I20260620 14:34:30.896010 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:34:30.898623 22397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:30.900003 22382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:34:30.900141 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "20279e36eb7f4756abe53e51d7572458"
format_stamp: "Formatted at 2026-06-20 14:34:30 on dist-test-slave-gqzk"
server_key: "e109a5522700bccbe5281e87b61055c0"
server_key_iv: "80d18891bb3722f5ef5f2574c10370a9"
server_key_version: "encryptionkey@0"
I20260620 14:34:30.900274 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:30.924736 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:30.927747 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:30.927954 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:30.934602 22382 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34905
I20260620 14:34:30.934628 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34905 every 8 connection(s)
I20260620 14:34:30.935564 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:30.938387 22450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:30.944100 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: Bootstrap starting.
I20260620 14:34:30.944522 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22382
I20260620 14:34:30.944643 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:30.944958 4329 external_mini_cluster.cc:1428] Setting key cb238f780d2a96e1cf0234ad9c3a7fea
I20260620 14:34:30.946868 22450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:30.947818 22450 log.cc:824] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:30.950196 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: No bootstrap required, opened a new log
I20260620 14:34:30.954228 22450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } }
I20260620 14:34:30.954469 22450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 20 14:34:30 dist-test-slave-gqzk krb5kdc[22369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:30.954512 22450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20279e36eb7f4756abe53e51d7572458, State: Initialized, Role: FOLLOWER
I20260620 14:34:30.954972 22450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } }
I20260620 14:34:30.955133 22450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:30.955194 22450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:30.955309 22450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:30.956478 22450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } }
I20260620 14:34:30.956903 22450 leader_election.cc:302] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20279e36eb7f4756abe53e51d7572458; no voters:
I20260620 14:34:30.957293 22450 leader_election.cc:288] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:30.957397 22455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:30.957607 22455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [term 1 LEADER]: Becoming Leader. State: Replica: 20279e36eb7f4756abe53e51d7572458, State: Running, Role: LEADER
I20260620 14:34:30.958097 22455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } }
I20260620 14:34:30.958624 22450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:30.959887 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20279e36eb7f4756abe53e51d7572458. Latest consensus state: current_term: 1 leader_uuid: "20279e36eb7f4756abe53e51d7572458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } } }
I20260620 14:34:30.959950 22456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20279e36eb7f4756abe53e51d7572458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20279e36eb7f4756abe53e51d7572458" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34905 } } }
I20260620 14:34:30.960007 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:30.960040 22456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:30.961186 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:30.946429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.4.58.126:34905)
0620 14:34:30.946918 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:30.946939 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:30.946984 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:34:30.947900 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:30.947927 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:30.948365 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:30.948695 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:30.949471 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:30.950925 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:30.951839 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:30.952544 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:30.952711 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:30.955077 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:30.955084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:30.955102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:30.955138 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:30.957217 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:30.957784 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:30.957791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:30.957798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:30.957964 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:30.958297 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:30.958300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:30.958302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:30.958815 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:30.958981 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:30.959453 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:34:30.959786 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":149,"threads_started":1}
W20260620 14:34:30.963596 22470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:30.963676 22470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:30.963729 22466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:30.964576 22466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:30.970697 22466 catalog_manager.cc:1347] Generated new cluster ID: 3b2106dc4d8f410b99eb42227f141b04
I20260620 14:34:30.970788 22466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:30.982951 22466 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:30.983974 22466 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:30.988488 22466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: Generated new TSK 0
I20260620 14:34:30.989279 22466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:34:31.018172 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:31.129405 22476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.129668 22476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.129710 22476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.133685 22476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.133751 22476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.133809 22476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:31.139132 22476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:31.140172 22476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.141204 22476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.151023 22481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.151050 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.151103 22476 server_base.cc:1034] running on GCE node
W20260620 14:34:31.151059 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.151639 22476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.152311 22476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.153477 22476 hybrid_clock.cc:648] HybridClock initialized: now 1781966071153456 us; error 40 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.161168 22476 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:34:31.162177 22476 webserver.cc:460] Webserver started at http://127.4.58.65:44077/ using document root <none> and password file <none>
I20260620 14:34:31.162755 22476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.162827 22476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.163038 22476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:31.165251 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "21f0cf46aaa14d0a976020485bf6d1e1"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "7f45ded73f3c1e0a9c0bad9b3d5a9c05"
server_key_iv: "dbe2113c3f88be8262a673f15e631b13"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.165791 22476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "21f0cf46aaa14d0a976020485bf6d1e1"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "7f45ded73f3c1e0a9c0bad9b3d5a9c05"
server_key_iv: "dbe2113c3f88be8262a673f15e631b13"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.169468 22476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:31.171975 22491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.173273 22476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:31.173427 22476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "21f0cf46aaa14d0a976020485bf6d1e1"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "7f45ded73f3c1e0a9c0bad9b3d5a9c05"
server_key_iv: "dbe2113c3f88be8262a673f15e631b13"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.173521 22476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.203225 22476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.206401 22476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.206547 22476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.207077 22476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.207861 22476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:31.207907 22476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.207947 22476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:31.207966 22476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.216197 22476 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:36881
I20260620 14:34:31.216234 22604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:36881 every 8 connection(s)
I20260620 14:34:31.217198 22476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:31.225951 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22476
I20260620 14:34:31.226111 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:31.226449 4329 external_mini_cluster.cc:1428] Setting key 556ff4fd15163420b62187b11770b62f
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:31.231231 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.219070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:39745 (local address 127.4.58.126:34905)
0620 14:34:31.219254 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.219257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.220077 (+ 820us) server_negotiation.cc:407] Connection header received
0620 14:34:31.221407 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.221409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.221457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.221531 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.222745 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.223268 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.224564 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.224860 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.224938 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.227735 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.227739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.227742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.227771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.229234 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.229729 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.229733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.229734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.229825 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.230139 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.230142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.230144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.230373 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.230453 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.230935 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.231066 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:34:31.232081 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.219451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34905 (local address 127.4.58.65:39745)
0620 14:34:31.219916 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.219963 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.221195 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.221709 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.221716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.222093 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.222608 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.222623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.223410 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.223413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.224435 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.224446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.224996 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.225000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.225203 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.225770 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.225792 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.227547 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.229357 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.229364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.229377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.229633 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.229926 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.229928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.229930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.230058 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.230456 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.230461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.230703 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.230869 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.231196 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260620 14:34:31.233489 22605 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34905
I20260620 14:34:31.233734 22605 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.234226 22605 heartbeater.cc:511] Master 127.4.58.126:34905 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:34:31.235834 22414 ts_manager.cc:194] Registered new tserver with Master: 21f0cf46aaa14d0a976020485bf6d1e1 (127.4.58.65:36881)
I20260620 14:34:31.237006 22414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:39745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:34:31.252288 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:31.363382 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.363651 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.363692 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.367715 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.367798 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.367856 22610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:31.373179 22610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:31.374284 22610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.375329 22610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.384905 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.384905 22615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.384955 22610 server_base.cc:1034] running on GCE node
W20260620 14:34:31.384902 22616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.385614 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.386243 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.387398 22610 hybrid_clock.cc:648] HybridClock initialized: now 1781966071387371 us; error 39 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.390115 22610 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:34:31.391062 22610 webserver.cc:460] Webserver started at http://127.4.58.66:34095/ using document root <none> and password file <none>
I20260620 14:34:31.391628 22610 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.391690 22610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.391921 22610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:31.394037 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "15584502bf694f43ba2147130464d867"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "91d82f4411258bcdc1d74370c8efc9cd"
server_key_iv: "bbcee0f28fe30326d8aa95966df1138d"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.394578 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "15584502bf694f43ba2147130464d867"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "91d82f4411258bcdc1d74370c8efc9cd"
server_key_iv: "bbcee0f28fe30326d8aa95966df1138d"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.398388 22610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:34:31.400878 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.402248 22610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:31.402354 22610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "15584502bf694f43ba2147130464d867"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "91d82f4411258bcdc1d74370c8efc9cd"
server_key_iv: "bbcee0f28fe30326d8aa95966df1138d"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.402449 22610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.421044 22610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.424199 22610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.424353 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.424922 22610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.425704 22610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:31.425758 22610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.425797 22610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:31.425817 22610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.434167 22610 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:46075
I20260620 14:34:31.434207 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:46075 every 8 connection(s)
I20260620 14:34:31.435041 22610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:31.438391 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22610
I20260620 14:34:31.438503 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:31.438822 4329 external_mini_cluster.cc:1428] Setting key bbf2056e3b0fa1e7ebfd695ae2c5e3e7
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:34:31.449520 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.436819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:48187 (local address 127.4.58.126:34905)
0620 14:34:31.436945 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.436948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.437762 (+ 814us) server_negotiation.cc:407] Connection header received
0620 14:34:31.439159 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.439163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.439224 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.439300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.440566 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.441339 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.442887 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.443114 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.443190 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.446012 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.446015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.446018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.446044 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.447373 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.447948 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.447950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.447952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.448019 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.448361 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.448363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.448364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.448544 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.448619 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.449279 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.449396 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:34:31.450371 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.437190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34905 (local address 127.4.58.66:48187)
0620 14:34:31.437614 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.437657 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.438949 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.439494 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.439502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.439860 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.440426 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.440442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.441458 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.441460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.442761 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.442773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.443260 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.443263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.443461 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.444048 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.444068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.445839 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.447486 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.447492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.447509 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.447864 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.448091 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.448094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.448096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.448279 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.448633 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.448638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.448958 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.449166 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.449461 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
I20260620 14:34:31.451602 22739 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34905
I20260620 14:34:31.451869 22739 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.452441 22739 heartbeater.cc:511] Master 127.4.58.126:34905 requested a full tablet report, sending...
I20260620 14:34:31.453569 22414 ts_manager.cc:194] Registered new tserver with Master: 15584502bf694f43ba2147130464d867 (127.4.58.66:46075)
I20260620 14:34:31.454133 22414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:48187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:34:31.465741 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:31.573019 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:31.573312 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:31.573361 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:31.577386 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:31.577442 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:31.577508 22744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:31.583037 22744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:31.584117 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:31.585175 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:31.595170 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:31.595160 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.595240 22744 server_base.cc:1034] running on GCE node
W20260620 14:34:31.595170 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:31.595890 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:31.596573 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:31.597731 22744 hybrid_clock.cc:648] HybridClock initialized: now 1781966071597713 us; error 53 us; skew 500 ppm
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:31.600507 22744 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:34:31.601461 22744 webserver.cc:460] Webserver started at http://127.4.58.67:41723/ using document root <none> and password file <none>
I20260620 14:34:31.602015 22744 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:31.602082 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:31.602294 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:31.604354 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "e025c44942ae4c56a248de0047baa1cb"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "92e0e0a73b2c32b08cd41822b17e474b"
server_key_iv: "92746d5dba3387d6f1f02d12a401f142"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.604919 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "e025c44942ae4c56a248de0047baa1cb"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "92e0e0a73b2c32b08cd41822b17e474b"
server_key_iv: "92746d5dba3387d6f1f02d12a401f142"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.608582 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:31.611019 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.612313 22744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:31.612428 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "e025c44942ae4c56a248de0047baa1cb"
format_stamp: "Formatted at 2026-06-20 14:34:31 on dist-test-slave-gqzk"
server_key: "92e0e0a73b2c32b08cd41822b17e474b"
server_key_iv: "92746d5dba3387d6f1f02d12a401f142"
server_key_version: "encryptionkey@0"
I20260620 14:34:31.612528 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:31.629060 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:31.632066 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:31.632239 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:31.632836 22744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:31.633642 22744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:31.633688 22744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.633741 22744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:31.633769 22744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:31.642472 22744 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:34013
I20260620 14:34:31.642508 22872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:34013 every 8 connection(s)
I20260620 14:34:31.643333 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:31.651935 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22744
I20260620 14:34:31.652084 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:31.652437 4329 external_mini_cluster.cc:1428] Setting key b8caca8d1106189aa6fe32089b546d61
Jun 20 14:34:31 dist-test-slave-gqzk krb5kdc[22369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966071, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:31.657058 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.645066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:49689 (local address 127.4.58.126:34905)
0620 14:34:31.645194 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.645198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.646080 (+ 882us) server_negotiation.cc:407] Connection header received
0620 14:34:31.647424 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.647427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.647477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.647551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.648778 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.649326 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.650519 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.650742 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.650840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.653505 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.653510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.653513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.653543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.655127 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.655648 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.655650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.655651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.655721 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.656057 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.656059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.656060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.656241 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.656313 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.656830 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.656935 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:34:31.657986 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.645434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34905 (local address 127.4.58.67:49689)
0620 14:34:31.645910 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.645957 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.647208 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.647722 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.647730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.648123 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:31.648628 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.648642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.649428 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.649430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.650416 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.650426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.650880 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.650882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.651036 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.651591 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:31.651608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:31.653339 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:31.655241 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.655247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.655261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.655569 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.655834 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:31.655836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:31.655838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:31.655991 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:31.656315 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:31.656320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:31.656581 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.656743 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.657085 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260620 14:34:31.659179 22873 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34905
I20260620 14:34:31.659430 22873 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:31.659933 22873 heartbeater.cc:511] Master 127.4.58.126:34905 requested a full tablet report, sending...
I20260620 14:34:31.661058 22414 ts_manager.cc:194] Registered new tserver with Master: e025c44942ae4c56a248de0047baa1cb (127.4.58.67:34013)
I20260620 14:34:31.661619 22414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:49689
I20260620 14:34:31.666788 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:31.676283 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.668808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.4.58.126:34905)
0620 14:34:31.668944 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.668948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.669082 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:34:31.669195 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.669197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.669247 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.669317 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:31.670008 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.670558 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.671468 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.671725 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.671823 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.673432 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:31.673436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:31.673438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:31.673466 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:31.674541 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.675040 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.675043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.675044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.675119 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:31.675425 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:31.675429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:31.675430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:31.675639 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:31.675774 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.676034 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.676129 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:34:31.679584 22414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:31.682798 22414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:31.697876 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.692177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46075 (local address 127.0.0.1:54994)
0620 14:34:31.693669 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.693681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.693764 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.694159 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.694162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.694180 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.694256 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.694261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.695367 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.695370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.696464 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.696474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697513 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.697595 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697614 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.697671 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.697724 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"mutex_wait_us":30,"thread_start_us":109,"threads_started":1}
I20260620 14:34:31.698020 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.691647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.4.58.67:34013)
0620 14:34:31.692882 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.692893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.692919 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:34:31.692995 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.693001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.693229 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.693421 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.693763 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.694628 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.696623 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697440 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697659 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.697764 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.697885 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":160,"threads_started":1}
I20260620 14:34:31.698210 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.692315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.4.58.65:36881)
0620 14:34:31.693001 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.693016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.693035 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:34:31.693103 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.693108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.693299 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.693442 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.693729 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.694866 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.696291 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697129 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697232 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697392 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.697945 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.698056 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":104,"threads_started":1}
I20260620 14:34:31.698616 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.692367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.4.58.66:46075)
0620 14:34:31.692832 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.692845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.693750 (+ 905us) server_negotiation.cc:407] Connection header received
0620 14:34:31.693859 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.693864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.694027 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.694162 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.694346 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.695248 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.696601 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697203 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697313 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697455 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.698396 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.698497 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":85,"threads_started":1}
I20260620 14:34:31.698009 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.691505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:34013 (local address 127.0.0.1:35992)
0620 14:34:31.691936 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.691984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.692104 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.693413 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.693420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.693443 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.693631 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.693639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.694737 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.694740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.696169 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.696175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697547 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.697639 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697660 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.697733 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.697807 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":88,"spinlock_wait_cycles":1280,"thread_start_us":107,"threads_started":1}
I20260620 14:34:31.697876 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.691597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36881 (local address 127.0.0.1:44910)
0620 14:34:31.692432 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.692445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.692529 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.693428 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.693431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.693460 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.693623 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.693629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.694993 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.694997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.696163 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.696174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.697269 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.697273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.697387 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.697426 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.697534 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.697660 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":6,"spinlock_wait_cycles":2304,"thread_start_us":114,"threads_started":1}
I20260620 14:34:31.701877 22673 tablet_service.cc:1467] Processing CreateTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 (DEFAULT_TABLE table=test-table [id=eb21f2cffe8f4cbaa45688de24de9ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.701906 22807 tablet_service.cc:1467] Processing CreateTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 (DEFAULT_TABLE table=test-table [id=eb21f2cffe8f4cbaa45688de24de9ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.701910 22539 tablet_service.cc:1467] Processing CreateTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 (DEFAULT_TABLE table=test-table [id=eb21f2cffe8f4cbaa45688de24de9ade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:31.702940 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35936eaf72514a0b921f9e4f4f49d4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.702941 22539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35936eaf72514a0b921f9e4f4f49d4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.702939 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35936eaf72514a0b921f9e4f4f49d4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:31.709952 22890 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Bootstrap starting.
I20260620 14:34:31.711264 22892 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Bootstrap starting.
I20260620 14:34:31.713101 22891 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: Bootstrap starting.
I20260620 14:34:31.713215 22890 tablet_bootstrap.cc:654] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.713517 22892 tablet_bootstrap.cc:654] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.714226 22890 log.cc:824] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.714752 22892 log.cc:824] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.715118 22891 tablet_bootstrap.cc:654] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:31.716185 22891 log.cc:824] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:31.716501 22890 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: No bootstrap required, opened a new log
I20260620 14:34:31.716728 22892 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: No bootstrap required, opened a new log
I20260620 14:34:31.716703 22890 ts_tablet_manager.cc:1397] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 14:34:31.716970 22892 ts_tablet_manager.cc:1397] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:31.717805 22891 tablet_bootstrap.cc:492] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: No bootstrap required, opened a new log
I20260620 14:34:31.717942 22891 ts_tablet_manager.cc:1397] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:34:31.720729 22891 raft_consensus.cc:348] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.720803 22890 raft_consensus.cc:348] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.720818 22892 raft_consensus.cc:348] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.721105 22890 raft_consensus.cc:374] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.721107 22892 raft_consensus.cc:374] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.721107 22891 raft_consensus.cc:374] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:31.721158 22891 raft_consensus.cc:729] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f0cf46aaa14d0a976020485bf6d1e1, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.721156 22892 raft_consensus.cc:729] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e025c44942ae4c56a248de0047baa1cb, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.721156 22890 raft_consensus.cc:729] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15584502bf694f43ba2147130464d867, State: Initialized, Role: FOLLOWER
I20260620 14:34:31.721642 22892 consensus_queue.cc:260] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.721678 22891 consensus_queue.cc:260] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.721951 22890 consensus_queue.cc:260] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.722414 22873 heartbeater.cc:503] Master 127.4.58.126:34905 was elected leader, sending a full tablet report...
I20260620 14:34:31.722659 22739 heartbeater.cc:503] Master 127.4.58.126:34905 was elected leader, sending a full tablet report...
I20260620 14:34:31.722867 22605 heartbeater.cc:503] Master 127.4.58.126:34905 was elected leader, sending a full tablet report...
I20260620 14:34:31.722903 22890 ts_tablet_manager.cc:1428] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:31.723901 22892 ts_tablet_manager.cc:1428] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:34:31.723901 22891 ts_tablet_manager.cc:1428] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 14:34:31.797899 22896 raft_consensus.cc:482] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:31.798115 22896 raft_consensus.cc:504] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.799291 22896 leader_election.cc:288] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f0cf46aaa14d0a976020485bf6d1e1 (127.4.58.65:36881), 15584502bf694f43ba2147130464d867 (127.4.58.66:46075)
I20260620 14:34:31.802724 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.799501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36881 (local address 127.4.58.67:48193)
0620 14:34:31.799609 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.799626 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.799720 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.800019 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.800022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.800040 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.800142 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.800148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.800869 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.800872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.801704 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.801714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.802326 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.802330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.802434 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.802452 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.802516 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.802586 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":37}
I20260620 14:34:31.802723 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.799692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:48193 (local address 127.4.58.65:36881)
0620 14:34:31.799826 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.799830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.799841 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:34:31.799889 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.799891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.799928 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.799991 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.800246 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.800732 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.801833 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.802213 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.802304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.802373 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.802543 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.802620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:34:31.803118 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.799690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:34815 (local address 127.4.58.66:46075)
0620 14:34:31.799877 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.799882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.800104 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:34:31.800206 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.800208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.800255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.800322 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:31.800579 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.801121 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.802202 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.802593 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.802685 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.802753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.802937 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.803017 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 14:34:31.803508 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.799584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46075 (local address 127.4.58.67:34815)
0620 14:34:31.800007 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.800018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:31.800120 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:31.800348 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:31.800351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:31.800366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:31.800477 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.800482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.801227 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.801230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.802088 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:31.802098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.802700 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.802704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:31.802796 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.802821 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:34:31.803281 (+ 460us) client_negotiation.cc:240] Negotiation successful
0620 14:34:31.803354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":2,"thread_start_us":128,"threads_started":1}
I20260620 14:34:31.803421 22559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35936eaf72514a0b921f9e4f4f49d4e7" candidate_uuid: "e025c44942ae4c56a248de0047baa1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" is_pre_election: true
I20260620 14:34:31.803785 22559 raft_consensus.cc:2454] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e025c44942ae4c56a248de0047baa1cb in term 0.
I20260620 14:34:31.804148 22693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35936eaf72514a0b921f9e4f4f49d4e7" candidate_uuid: "e025c44942ae4c56a248de0047baa1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15584502bf694f43ba2147130464d867" is_pre_election: true
I20260620 14:34:31.804255 22763 leader_election.cc:302] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f0cf46aaa14d0a976020485bf6d1e1, e025c44942ae4c56a248de0047baa1cb; no voters:
I20260620 14:34:31.804445 22693 raft_consensus.cc:2454] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e025c44942ae4c56a248de0047baa1cb in term 0.
I20260620 14:34:31.804562 22896 raft_consensus.cc:2781] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:31.804649 22896 raft_consensus.cc:482] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:31.804690 22896 raft_consensus.cc:3037] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.805790 22896 raft_consensus.cc:504] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.806128 22896 leader_election.cc:288] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [CANDIDATE]: Term 1 election: Requested vote from peers 21f0cf46aaa14d0a976020485bf6d1e1 (127.4.58.65:36881), 15584502bf694f43ba2147130464d867 (127.4.58.66:46075)
I20260620 14:34:31.806434 22693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35936eaf72514a0b921f9e4f4f49d4e7" candidate_uuid: "e025c44942ae4c56a248de0047baa1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15584502bf694f43ba2147130464d867"
I20260620 14:34:31.806475 22559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35936eaf72514a0b921f9e4f4f49d4e7" candidate_uuid: "e025c44942ae4c56a248de0047baa1cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f0cf46aaa14d0a976020485bf6d1e1"
I20260620 14:34:31.806538 22693 raft_consensus.cc:3037] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.806607 22559 raft_consensus.cc:3037] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:31.807662 22693 raft_consensus.cc:2454] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e025c44942ae4c56a248de0047baa1cb in term 1.
I20260620 14:34:31.807703 22559 raft_consensus.cc:2454] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e025c44942ae4c56a248de0047baa1cb in term 1.
I20260620 14:34:31.807938 22760 leader_election.cc:302] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15584502bf694f43ba2147130464d867, e025c44942ae4c56a248de0047baa1cb; no voters:
I20260620 14:34:31.808108 22896 raft_consensus.cc:2781] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:31.808300 22896 raft_consensus.cc:686] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 1 LEADER]: Becoming Leader. State: Replica: e025c44942ae4c56a248de0047baa1cb, State: Running, Role: LEADER
I20260620 14:34:31.808595 22896 consensus_queue.cc:237] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } }
I20260620 14:34:31.811718 22412 catalog_manager.cc:5495] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb reported cstate change: term changed from 0 to 1, leader changed from <none> to e025c44942ae4c56a248de0047baa1cb (127.4.58.67). New cstate: current_term: 1 leader_uuid: "e025c44942ae4c56a248de0047baa1cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e025c44942ae4c56a248de0047baa1cb" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 34013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 } health_report { overall_health: UNKNOWN } } }
I20260620 14:34:31.851536 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.848336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.4.58.67:34013)
0620 14:34:31.848472 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:31.848476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:31.848491 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:34:31.848559 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:31.848561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:31.848614 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:31.848702 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:31.848942 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.849450 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.850226 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:31.850438 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:31.850516 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:31.850765 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:31.851158 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:31.851270 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:31.851325 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:34:31.851364 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:34:31.857668 22693 raft_consensus.cc:1261] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 1 FOLLOWER]: Refusing update from remote peer e025c44942ae4c56a248de0047baa1cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:31.857668 22559 raft_consensus.cc:1261] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 1 FOLLOWER]: Refusing update from remote peer e025c44942ae4c56a248de0047baa1cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:31.858412 22896 consensus_queue.cc:1035] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "15584502bf694f43ba2147130464d867" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:31.858637 22900 consensus_queue.cc:1035] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f0cf46aaa14d0a976020485bf6d1e1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:31.870153 22905 mvcc.cc:205] Tried to move back new op lower bound from 7298933030320762880 to 7298933030130835456. Current Snapshot: MvccSnapshot[applied={T|T < 7298933030320762880 or (T in {7298933030320762880})}]
I20260620 14:34:31.874734 22412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:31.874913 22412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:31.877306 22412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20279e36eb7f4756abe53e51d7572458: Sending DeleteTablet for 3 replicas of tablet 35936eaf72514a0b921f9e4f4f49d4e7
I20260620 14:34:31.878208 22539 tablet_service.cc:1514] Processing DeleteTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:44910
I20260620 14:34:31.878208 22807 tablet_service.cc:1514] Processing DeleteTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:35992
I20260620 14:34:31.878723 22912 tablet_replica.cc:330] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: stopping tablet replica
I20260620 14:34:31.878888 22913 tablet_replica.cc:330] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1: stopping tablet replica
I20260620 14:34:31.878849 22673 tablet_service.cc:1514] Processing DeleteTablet for tablet 35936eaf72514a0b921f9e4f4f49d4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:31 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:54994
I20260620 14:34:31.879119 22912 raft_consensus.cc:2229] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:31.879179 22913 raft_consensus.cc:2229] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:31.879474 22912 raft_consensus.cc:2258] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.879503 22913 raft_consensus.cc:2258] T 35936eaf72514a0b921f9e4f4f49d4e7 P 21f0cf46aaa14d0a976020485bf6d1e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:31.879801 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22476
I20260620 14:34:31.881465 22912 ts_tablet_manager.cc:1905] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:31.883621 22912 ts_tablet_manager.cc:1918] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:31.883690 22912 log.cc:1196] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/35936eaf72514a0b921f9e4f4f49d4e7
I20260620 14:34:31.883903 22912 ts_tablet_manager.cc:1939] T 35936eaf72514a0b921f9e4f4f49d4e7 P e025c44942ae4c56a248de0047baa1cb: Deleting consensus metadata
I20260620 14:34:31.885129 22400 catalog_manager.cc:4841] TS e025c44942ae4c56a248de0047baa1cb (127.4.58.67:34013): tablet 35936eaf72514a0b921f9e4f4f49d4e7 (table test-table [id=eb21f2cffe8f4cbaa45688de24de9ade]) successfully deleted
I20260620 14:34:31.886328 22914 tablet_replica.cc:330] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: stopping tablet replica
I20260620 14:34:31.886651 22914 raft_consensus.cc:2229] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:31.887006 22914 raft_consensus.cc:2258] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:34:31.887216 22401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:36881 (error 108)
I20260620 14:34:31.888236 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.887947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:36881 (local address 127.0.0.1:44926)
0620 14:34:31.888079 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.888149 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:34:31.888376 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22610
I20260620 14:34:31.888598 22914 ts_tablet_manager.cc:1905] T 35936eaf72514a0b921f9e4f4f49d4e7 P 15584502bf694f43ba2147130464d867: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:34:31.891366 22401 catalog_manager.cc:4584] TS 21f0cf46aaa14d0a976020485bf6d1e1 (127.4.58.65:36881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35936eaf72514a0b921f9e4f4f49d4e7: Network error: Client connection negotiation failed: client connection to 127.4.58.65:36881: connect: Connection refused (error 111)
I20260620 14:34:31.895345 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22744
I20260620 14:34:31.895794 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:31.895490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:46075 (local address 127.0.0.1:54998)
0620 14:34:31.895659 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:31.895702 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:46075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:34:31.900454 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22382
2026-06-20T14:34:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2236 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:31Z Disabled control of system clock
I20260620 14:34:31.929850 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36673
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:32.042838 22919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:32.043124 22919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:32.043160 22919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:32.047251 22919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:32.047314 22919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:32.047328 22919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:32.047344 22919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:32.047359 22919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:32.052253 22919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:36673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:32.053390 22919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:32.054826 22919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:32.063582 22924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.063630 22919 server_base.cc:1034] running on GCE node
W20260620 14:34:32.063589 22927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:32.063589 22925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.064431 22919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:32.065658 22919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:32.066829 22919 hybrid_clock.cc:648] HybridClock initialized: now 1781966072066802 us; error 41 us; skew 500 ppm
I20260620 14:34:32.068686 22919 webserver.cc:460] Webserver started at http://127.4.58.126:43855/ using document root <none> and password file <none>
I20260620 14:34:32.069286 22919 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:32.069348 22919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:32.069515 22919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:32.071529 22919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "59eaa59657e747ed900667c3cd3ab01a"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "e9a037c74bbd0795fab7503d58f157d3"
server_key_iv: "a84c08db118ee2949cf500c124892182"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.072032 22919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "59eaa59657e747ed900667c3cd3ab01a"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "e9a037c74bbd0795fab7503d58f157d3"
server_key_iv: "a84c08db118ee2949cf500c124892182"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.075680 22919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:34:32.078105 22933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.079339 22919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:32.079442 22919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "59eaa59657e747ed900667c3cd3ab01a"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "e9a037c74bbd0795fab7503d58f157d3"
server_key_iv: "a84c08db118ee2949cf500c124892182"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.079527 22919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:32.102064 22919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:32.102790 22919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:32.102950 22919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:32.109895 22919 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:36673
I20260620 14:34:32.109911 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:36673 every 8 connection(s)
I20260620 14:34:32.110879 22919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:32.114038 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:32.115724 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 22919
I20260620 14:34:32.115828 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:32.116128 4329 external_mini_cluster.cc:1428] Setting key c38a1ded61972dbfd09d7a1772db7df9
I20260620 14:34:32.120054 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: Bootstrap starting.
I20260620 14:34:32.122318 22986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:32.123055 22986 log.cc:824] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:32.125427 22986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: No bootstrap required, opened a new log
I20260620 14:34:32.125947 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.117592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.4.58.126:36673)
0620 14:34:32.118344 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.118361 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.118397 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:34:32.119203 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.119223 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.119646 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.119972 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:32.120413 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.121489 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.122402 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.122869 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.123011 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.123864 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:32.123871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:32.123889 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:32.124370 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:32.124446 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:32.124471 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:32.124568 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.124718 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.125013 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":138,"threads_started":1}
I20260620 14:34:32.129335 22986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } }
I20260620 14:34:32.129551 22986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:32.129583 22986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eaa59657e747ed900667c3cd3ab01a, State: Initialized, Role: FOLLOWER
I20260620 14:34:32.129933 22986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } }
I20260620 14:34:32.130040 22986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:32.130084 22986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:32.130149 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:32.130995 22986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } }
I20260620 14:34:32.131297 22986 leader_election.cc:302] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59eaa59657e747ed900667c3cd3ab01a; no voters:
I20260620 14:34:32.131549 22986 leader_election.cc:288] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:32.131699 22991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:32.131915 22991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [term 1 LEADER]: Becoming Leader. State: Replica: 59eaa59657e747ed900667c3cd3ab01a, State: Running, Role: LEADER
I20260620 14:34:32.132205 22991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } }
I20260620 14:34:32.132686 22986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:32.133745 22992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59eaa59657e747ed900667c3cd3ab01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } } }
I20260620 14:34:32.133876 22992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:32.134234 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59eaa59657e747ed900667c3cd3ab01a. Latest consensus state: current_term: 1 leader_uuid: "59eaa59657e747ed900667c3cd3ab01a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaa59657e747ed900667c3cd3ab01a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 36673 } } }
I20260620 14:34:32.134403 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:32.134589 23000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:32.137380 23000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:32.142822 23000 catalog_manager.cc:1347] Generated new cluster ID: faee5024b66c47a28f38fd9b88c9de92
I20260620 14:34:32.142900 23000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:32.167416 23000 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:32.168411 23000 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:32.179188 23000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: Generated new TSK 0
I20260620 14:34:32.179896 23000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:32.187041 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--builtin_ntp_servers=127.4.58.84:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:32.296329 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:32.296597 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:32.296634 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:32.301096 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:32.301177 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:32.301249 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:32.307217 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:32.308322 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:32.309785 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:32.319293 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:32.319378 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.319447 23010 server_base.cc:1034] running on GCE node
W20260620 14:34:32.319378 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.319999 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:32.320608 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:32.321832 23010 hybrid_clock.cc:648] HybridClock initialized: now 1781966072321808 us; error 40 us; skew 500 ppm
I20260620 14:34:32.323745 23010 webserver.cc:460] Webserver started at http://127.4.58.65:35097/ using document root <none> and password file <none>
I20260620 14:34:32.324308 23010 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:32.324374 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:32.324561 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:32.326642 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "cce523df3a084215a2ff20377bd20544"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "37a26df3dbc4f3944995f798e4ad4caf"
server_key_iv: "4a52f29729e287c63b9a56d241549994"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.327176 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "cce523df3a084215a2ff20377bd20544"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "37a26df3dbc4f3944995f798e4ad4caf"
server_key_iv: "4a52f29729e287c63b9a56d241549994"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.330801 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:34:32.333348 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.334667 23010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:32.334777 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "cce523df3a084215a2ff20377bd20544"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "37a26df3dbc4f3944995f798e4ad4caf"
server_key_iv: "4a52f29729e287c63b9a56d241549994"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.334877 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:32.353055 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:32.353720 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:32.353889 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:32.354465 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:32.355413 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:32.355459 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.355510 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:32.355538 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.364228 23010 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:42919
I20260620 14:34:32.364250 23137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:42919 every 8 connection(s)
I20260620 14:34:32.365204 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:32.373314 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23010
I20260620 14:34:32.373458 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:32.373822 4329 external_mini_cluster.cc:1428] Setting key 1d8847d9f1eed9be63bfddb2ce876685
I20260620 14:34:32.374559 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.367014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:54545 (local address 127.4.58.126:36673)
0620 14:34:32.367272 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.367277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.368375 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 14:34:32.369740 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.369743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.369798 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.369885 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:32.370853 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.371366 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.372564 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.372824 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.372899 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.374105 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:32.374108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:32.374110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:32.374122 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:32.374141 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:32.374149 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:32.374204 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.374357 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.374421 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260620 14:34:32.375430 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.367481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36673 (local address 127.4.58.65:54545)
0620 14:34:32.368208 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.368260 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.369551 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.370091 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.370098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.370226 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:32.370731 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.370744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.371483 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.371486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.372463 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.372472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.372947 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.372950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.373187 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.373803 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:32.373805 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:32.373860 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:32.373947 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:32.374224 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:32.374234 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:32.374239 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.374339 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.374570 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260620 14:34:32.375887 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--builtin_ntp_servers=127.4.58.84:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:32.376626 23138 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36673
I20260620 14:34:32.376896 23138 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:32.377460 23138 heartbeater.cc:511] Master 127.4.58.126:36673 requested a full tablet report, sending...
I20260620 14:34:32.379062 22950 ts_manager.cc:194] Registered new tserver with Master: cce523df3a084215a2ff20377bd20544 (127.4.58.65:42919)
I20260620 14:34:32.380658 22950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.65:54545
W20260620 14:34:32.485630 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:32.485895 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:32.485934 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:32.489948 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:32.490013 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:32.490069 23141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:32.495215 23141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:32.496279 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:32.497629 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:32.506914 23141 server_base.cc:1034] running on GCE node
W20260620 14:34:32.506865 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:32.506865 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:32.506862 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.507601 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:32.508232 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:32.509399 23141 hybrid_clock.cc:648] HybridClock initialized: now 1781966072509380 us; error 31 us; skew 500 ppm
I20260620 14:34:32.511130 23141 webserver.cc:460] Webserver started at http://127.4.58.66:35709/ using document root <none> and password file <none>
I20260620 14:34:32.511719 23141 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:32.511790 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:32.511978 23141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:32.514072 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "d755b2f30d2b42cbbf6297746216a211"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "948495c3c3c137337761a1bea85ebc6d"
server_key_iv: "268c4dbb89a42ba377d5e4cd7914dec7"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.514605 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "d755b2f30d2b42cbbf6297746216a211"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "948495c3c3c137337761a1bea85ebc6d"
server_key_iv: "268c4dbb89a42ba377d5e4cd7914dec7"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.518462 23141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:34:32.520835 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.522042 23141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:32.522158 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "d755b2f30d2b42cbbf6297746216a211"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "948495c3c3c137337761a1bea85ebc6d"
server_key_iv: "268c4dbb89a42ba377d5e4cd7914dec7"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.522254 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:32.536494 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:32.537195 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:32.537364 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:32.537952 23141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:32.538863 23141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:32.538913 23141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.538964 23141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:32.538996 23141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.547307 23141 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:41735
I20260620 14:34:32.547392 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:41735 every 8 connection(s)
I20260620 14:34:32.548215 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:32.552629 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23141
I20260620 14:34:32.552788 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:32.553136 4329 external_mini_cluster.cc:1428] Setting key beaebfe9e9eb1d195d4b8b9482749647
I20260620 14:34:32.555097 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--builtin_ntp_servers=127.4.58.84:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:32.558317 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.550173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:42207 (local address 127.4.58.126:36673)
0620 14:34:32.550344 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.550349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.551461 (+ 1112us) server_negotiation.cc:407] Connection header received
0620 14:34:32.552999 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.553001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.553052 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.553127 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:32.554189 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.554829 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.556364 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.556692 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.556866 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.557871 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:32.557875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:32.557878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:32.557892 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:32.557915 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:32.557924 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:32.557998 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.558114 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.558173 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:34:32.559190 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.550637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36673 (local address 127.4.58.66:42207)
0620 14:34:32.551319 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.551361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.552700 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.553366 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.553376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.553483 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:32.554061 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.554074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.554958 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.554962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.556230 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.556241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.556898 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.556901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.557041 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.557605 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:32.557607 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:32.557649 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:32.557713 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:32.558001 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:32.558006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:32.558010 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.558101 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.558327 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260620 14:34:32.560257 23269 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36673
I20260620 14:34:32.560521 23269 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:32.561198 23269 heartbeater.cc:511] Master 127.4.58.126:36673 requested a full tablet report, sending...
I20260620 14:34:32.562410 22950 ts_manager.cc:194] Registered new tserver with Master: d755b2f30d2b42cbbf6297746216a211 (127.4.58.66:41735)
I20260620 14:34:32.563005 22950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.66:42207
W20260620 14:34:32.665453 23272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:32.665746 23272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:32.665791 23272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:32.669862 23272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:32.669925 23272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:32.669991 23272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:32.674863 23272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:36673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:32.675977 23272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:32.677387 23272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:32.686815 23280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.686857 23272 server_base.cc:1034] running on GCE node
W20260620 14:34:32.686791 23277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:32.686802 23278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:32.687500 23272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:32.688100 23272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:32.689248 23272 hybrid_clock.cc:648] HybridClock initialized: now 1781966072689231 us; error 30 us; skew 500 ppm
I20260620 14:34:32.691067 23272 webserver.cc:460] Webserver started at http://127.4.58.67:45875/ using document root <none> and password file <none>
I20260620 14:34:32.691593 23272 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:32.691650 23272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:32.691817 23272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:32.693842 23272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "26a970e2e63e481089f701124e4e2ee3"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "ea1ae63f8ece16a3d8a665c80ea31ecd"
server_key_iv: "0413206e025f8c792c7e22cef9458b8a"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.694366 23272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "26a970e2e63e481089f701124e4e2ee3"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "ea1ae63f8ece16a3d8a665c80ea31ecd"
server_key_iv: "0413206e025f8c792c7e22cef9458b8a"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.698065 23272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:32.700626 23286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.701983 23272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:32.702090 23272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "26a970e2e63e481089f701124e4e2ee3"
format_stamp: "Formatted at 2026-06-20 14:34:32 on dist-test-slave-gqzk"
server_key: "ea1ae63f8ece16a3d8a665c80ea31ecd"
server_key_iv: "0413206e025f8c792c7e22cef9458b8a"
server_key_version: "encryptionkey@0"
I20260620 14:34:32.702222 23272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:32.722209 23272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:32.722908 23272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:32.723078 23272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:32.723661 23272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:32.724640 23272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:32.724685 23272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.724726 23272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:32.724826 23272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:32.732962 23272 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:37641
I20260620 14:34:32.732986 23399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:37641 every 8 connection(s)
I20260620 14:34:32.733933 23272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:32.741755 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23272
I20260620 14:34:32.741880 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:32.742162 4329 external_mini_cluster.cc:1428] Setting key c030cc15a4e43c89f28c4fe2248934e7
I20260620 14:34:32.742796 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.735696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:52709 (local address 127.4.58.126:36673)
0620 14:34:32.735845 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.735849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.736959 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 14:34:32.738324 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.738327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.738384 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.738455 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:32.739313 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.739859 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.741010 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.741217 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.741293 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.742335 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:32.742338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:32.742341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:32.742355 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:32.742377 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:32.742387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:32.742459 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.742597 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.742656 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:34:32.743892 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.736127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:36673 (local address 127.4.58.67:52709)
0620 14:34:32.736825 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.736866 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.738114 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.738604 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.738610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.738692 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:32.739192 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.739204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.739975 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.739978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.740922 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.740932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.741330 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.741333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.741476 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.742079 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:32.742081 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:32.742121 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:32.742184 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:32.742458 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:32.742466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:32.742472 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.742595 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.742890 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":127,"threads_started":1}
I20260620 14:34:32.744889 23400 heartbeater.cc:347] Connected to a master server at 127.4.58.126:36673
I20260620 14:34:32.745093 23400 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:32.745551 23400 heartbeater.cc:511] Master 127.4.58.126:36673 requested a full tablet report, sending...
I20260620 14:34:32.746455 22950 ts_manager.cc:194] Registered new tserver with Master: 26a970e2e63e481089f701124e4e2ee3 (127.4.58.67:37641)
I20260620 14:34:32.746971 22950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.67:52709
I20260620 14:34:32.756313 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:32.762120 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.758315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.4.58.126:36673)
0620 14:34:32.758451 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.758456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.758576 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:34:32.758695 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.758699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.758741 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.758803 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:32.759119 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.759653 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.760480 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.760693 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.760796 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.761640 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:32.761643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:32.761645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:32.761656 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:32.761673 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:32.761680 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:32.761776 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.761940 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.761983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:34:32.764871 22950 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:32.767796 22950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:32.783450 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.777331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:41735 (local address 127.0.0.1:56524)
0620 14:34:32.778068 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.778082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.778199 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.778766 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.778773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.778795 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:32.778942 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.778950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780297 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.780301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.781608 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.781616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.782923 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.782927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.783035 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.783077 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.783190 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.783327 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":102,"threads_started":1}
I20260620 14:34:32.783918 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.776895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42919 (local address 127.0.0.1:51912)
0620 14:34:32.777700 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.777752 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.777905 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.779312 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.779315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.779333 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:32.779473 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.779480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780950 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.780953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.782080 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.782092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.783542 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.783546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.783639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.783663 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.783710 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.783779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":24,"spinlock_wait_cycles":1408,"thread_start_us":79,"threads_started":1}
I20260620 14:34:32.783947 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.777005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.4.58.66:41735)
0620 14:34:32.777930 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.777940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.778298 (+ 358us) server_negotiation.cc:407] Connection header received
0620 14:34:32.778383 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.778388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.778593 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.778757 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:32.779082 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780156 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.781737 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.782479 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.782652 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.782865 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.783678 (+ 813us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.783790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":133,"threads_started":1}
I20260620 14:34:32.784703 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.777955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.4.58.67:37641)
0620 14:34:32.778260 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.778271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.778795 (+ 524us) server_negotiation.cc:407] Connection header received
0620 14:34:32.778878 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.778883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.779078 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.779353 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:32.779557 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780722 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.783330 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.784122 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.784227 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.784408 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.784455 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.784545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":125,"threads_started":1}
I20260620 14:34:32.786751 23203 tablet_service.cc:1467] Processing CreateTablet for tablet aff0476f94f44779b40aee73d626dd52 (DEFAULT_TABLE table=test-table [id=7d742e60f9be4e68a3d4ddc185a4b322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:32.787734 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff0476f94f44779b40aee73d626dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:32.784498 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.777563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:37641 (local address 127.0.0.1:51412)
0620 14:34:32.778513 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.778525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.778617 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.779346 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.779349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.779371 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:32.779460 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.779466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780884 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.780887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.783187 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.783198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.784223 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.784226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.784292 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.784310 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.784359 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.784406 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":26,"thread_start_us":70,"threads_started":1}
I20260620 14:34:32.789614 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.777027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.4.58.65:42919)
0620 14:34:32.777714 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.777736 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.778936 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 14:34:32.779010 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.779014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.779174 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.779614 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:32.779749 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.780845 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.782553 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.783420 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.789074 (+ 5654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.789302 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.789362 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.789473 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":81,"threads_started":1}
I20260620 14:34:32.791667 23334 tablet_service.cc:1467] Processing CreateTablet for tablet aff0476f94f44779b40aee73d626dd52 (DEFAULT_TABLE table=test-table [id=7d742e60f9be4e68a3d4ddc185a4b322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:32.792567 23334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff0476f94f44779b40aee73d626dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:32.792387 23072 tablet_service.cc:1467] Processing CreateTablet for tablet aff0476f94f44779b40aee73d626dd52 (DEFAULT_TABLE table=test-table [id=7d742e60f9be4e68a3d4ddc185a4b322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:32.793423 23072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff0476f94f44779b40aee73d626dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:32.795095 23417 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Bootstrap starting.
I20260620 14:34:32.797420 23417 tablet_bootstrap.cc:654] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:32.798313 23417 log.cc:824] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:32.799938 23419 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Bootstrap starting.
I20260620 14:34:32.800438 23420 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Bootstrap starting.
I20260620 14:34:32.800650 23417 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: No bootstrap required, opened a new log
I20260620 14:34:32.800889 23417 ts_tablet_manager.cc:1397] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:32.801949 23419 tablet_bootstrap.cc:654] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:32.802591 23420 tablet_bootstrap.cc:654] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:32.802628 23419 log.cc:824] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:32.803388 23420 log.cc:824] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:32.804369 23419 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: No bootstrap required, opened a new log
I20260620 14:34:32.804536 23419 ts_tablet_manager.cc:1397] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:34:32.805042 23417 raft_consensus.cc:348] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.805334 23417 raft_consensus.cc:374] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:32.805348 23420 tablet_bootstrap.cc:492] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: No bootstrap required, opened a new log
I20260620 14:34:32.805426 23417 raft_consensus.cc:729] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d755b2f30d2b42cbbf6297746216a211, State: Initialized, Role: FOLLOWER
I20260620 14:34:32.805514 23420 ts_tablet_manager.cc:1397] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:34:32.805908 23417 consensus_queue.cc:260] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.806881 23417 ts_tablet_manager.cc:1428] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:32.807206 23269 heartbeater.cc:503] Master 127.4.58.126:36673 was elected leader, sending a full tablet report...
I20260620 14:34:32.808215 23419 raft_consensus.cc:348] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.808473 23419 raft_consensus.cc:374] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:32.808557 23419 raft_consensus.cc:729] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a970e2e63e481089f701124e4e2ee3, State: Initialized, Role: FOLLOWER
I20260620 14:34:32.808990 23419 consensus_queue.cc:260] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.809644 23400 heartbeater.cc:503] Master 127.4.58.126:36673 was elected leader, sending a full tablet report...
I20260620 14:34:32.809962 23419 ts_tablet_manager.cc:1428] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:34:32.810154 23420 raft_consensus.cc:348] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.810386 23420 raft_consensus.cc:374] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:32.810431 23420 raft_consensus.cc:729] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce523df3a084215a2ff20377bd20544, State: Initialized, Role: FOLLOWER
I20260620 14:34:32.810873 23420 consensus_queue.cc:260] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.811507 23138 heartbeater.cc:503] Master 127.4.58.126:36673 was elected leader, sending a full tablet report...
I20260620 14:34:32.811826 23420 ts_tablet_manager.cc:1428] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 14:34:32.866463 23139 tablet.cc:2369] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:32.931658 23423 raft_consensus.cc:482] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:32.931908 23423 raft_consensus.cc:504] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.933444 23423 leader_election.cc:288] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cce523df3a084215a2ff20377bd20544 (127.4.58.65:42919), 26a970e2e63e481089f701124e4e2ee3 (127.4.58.67:37641)
I20260620 14:34:32.937044 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.933781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:59971 (local address 127.4.58.65:42919)
0620 14:34:32.933915 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.933920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.933937 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:34:32.934106 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.934109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.934164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.934248 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:32.934573 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.935085 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936104 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.936491 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.936644 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.936849 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.936918 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:34:32.937043 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.933668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42919 (local address 127.4.58.66:59971)
0620 14:34:32.933865 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.933881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.933980 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.934267 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.934270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.934284 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:32.934441 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.934447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.935190 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.935193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.935987 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.935996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.936598 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.936695 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.936738 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.936835 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.936908 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":5}
I20260620 14:34:32.937137 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.933838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:49435 (local address 127.4.58.67:37641)
0620 14:34:32.934052 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.934056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.934144 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:34:32.934248 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.934251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.934293 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.934363 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:32.934669 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.935192 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936197 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.936576 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936667 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.936726 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.936953 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.937020 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 14:34:32.937191 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.933707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:37641 (local address 127.4.58.66:49435)
0620 14:34:32.934064 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:32.934075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:32.934157 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:32.934428 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:32.934432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:32.934451 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:32.934559 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.934563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.935291 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.935295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.936071 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:32.936079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.936691 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.936695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:32.936844 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.936862 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:34:32.936927 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:34:32.936993 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":33,"thread_start_us":130,"threads_started":1}
I20260620 14:34:32.937893 23092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff0476f94f44779b40aee73d626dd52" candidate_uuid: "d755b2f30d2b42cbbf6297746216a211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce523df3a084215a2ff20377bd20544" is_pre_election: true
I20260620 14:34:32.937906 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff0476f94f44779b40aee73d626dd52" candidate_uuid: "d755b2f30d2b42cbbf6297746216a211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a970e2e63e481089f701124e4e2ee3" is_pre_election: true
I20260620 14:34:32.938194 23092 raft_consensus.cc:2454] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d755b2f30d2b42cbbf6297746216a211 in term 0.
I20260620 14:34:32.938190 23354 raft_consensus.cc:2454] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d755b2f30d2b42cbbf6297746216a211 in term 0.
I20260620 14:34:32.938661 23159 leader_election.cc:302] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26a970e2e63e481089f701124e4e2ee3, cce523df3a084215a2ff20377bd20544, d755b2f30d2b42cbbf6297746216a211; no voters:
I20260620 14:34:32.938923 23423 raft_consensus.cc:2781] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:32.938997 23423 raft_consensus.cc:482] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:32.939025 23423 raft_consensus.cc:3037] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:32.940085 23423 raft_consensus.cc:504] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.940419 23423 leader_election.cc:288] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [CANDIDATE]: Term 1 election: Requested vote from peers cce523df3a084215a2ff20377bd20544 (127.4.58.65:42919), 26a970e2e63e481089f701124e4e2ee3 (127.4.58.67:37641)
I20260620 14:34:32.940797 23092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff0476f94f44779b40aee73d626dd52" candidate_uuid: "d755b2f30d2b42cbbf6297746216a211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cce523df3a084215a2ff20377bd20544"
I20260620 14:34:32.940824 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff0476f94f44779b40aee73d626dd52" candidate_uuid: "d755b2f30d2b42cbbf6297746216a211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26a970e2e63e481089f701124e4e2ee3"
I20260620 14:34:32.940939 23354 raft_consensus.cc:3037] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:32.940940 23092 raft_consensus.cc:3037] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:32.942214 23092 raft_consensus.cc:2454] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d755b2f30d2b42cbbf6297746216a211 in term 1.
I20260620 14:34:32.942214 23354 raft_consensus.cc:2454] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d755b2f30d2b42cbbf6297746216a211 in term 1.
I20260620 14:34:32.942549 23156 leader_election.cc:302] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26a970e2e63e481089f701124e4e2ee3, cce523df3a084215a2ff20377bd20544, d755b2f30d2b42cbbf6297746216a211; no voters:
I20260620 14:34:32.942721 23423 raft_consensus.cc:2781] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:32.942941 23423 raft_consensus.cc:686] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 1 LEADER]: Becoming Leader. State: Replica: d755b2f30d2b42cbbf6297746216a211, State: Running, Role: LEADER
I20260620 14:34:32.943315 23423 consensus_queue.cc:237] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } }
I20260620 14:34:32.946616 22949 catalog_manager.cc:5495] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 reported cstate change: term changed from 0 to 1, leader changed from <none> to d755b2f30d2b42cbbf6297746216a211 (127.4.58.66). New cstate: current_term: 1 leader_uuid: "d755b2f30d2b42cbbf6297746216a211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d755b2f30d2b42cbbf6297746216a211" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 } health_report { overall_health: UNKNOWN } } }
I20260620 14:34:32.974475 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:32.970910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.4.58.66:41735)
0620 14:34:32.971027 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:32.971032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:32.971047 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:34:32.971173 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:32.971177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:32.971247 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:32.971370 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:32.971643 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.972209 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.973128 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:32.973365 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:32.973449 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:32.973716 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:32.974132 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:32.974246 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:32.974307 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:34:32.974342 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:34:32.980324 23354 raft_consensus.cc:1261] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 1 FOLLOWER]: Refusing update from remote peer d755b2f30d2b42cbbf6297746216a211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:32.980324 23092 raft_consensus.cc:1261] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 1 FOLLOWER]: Refusing update from remote peer d755b2f30d2b42cbbf6297746216a211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:32.981035 23427 consensus_queue.cc:1035] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cce523df3a084215a2ff20377bd20544" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:32.981204 23423 consensus_queue.cc:1035] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a970e2e63e481089f701124e4e2ee3" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 37641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:34:32.985606 23401 tablet.cc:2369] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:32.986992 23430 mvcc.cc:205] Tried to move back new op lower bound from 7298933034920337408 to 7298933034778386432. Current Snapshot: MvccSnapshot[applied={T|T < 7298933034920337408}]
I20260620 14:34:32.988991 23431 mvcc.cc:205] Tried to move back new op lower bound from 7298933034920337408 to 7298933034778386432. Current Snapshot: MvccSnapshot[applied={T|T < 7298933034920337408}]
I20260620 14:34:32.998526 22950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:32.998689 22950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:33.001168 22950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59eaa59657e747ed900667c3cd3ab01a: Sending DeleteTablet for 3 replicas of tablet aff0476f94f44779b40aee73d626dd52
I20260620 14:34:33.001968 23072 tablet_service.cc:1514] Processing DeleteTablet for tablet aff0476f94f44779b40aee73d626dd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:32 UTC) from {username='slave'} at 127.0.0.1:51912
I20260620 14:34:33.002040 23203 tablet_service.cc:1514] Processing DeleteTablet for tablet aff0476f94f44779b40aee73d626dd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:32 UTC) from {username='slave'} at 127.0.0.1:56524
I20260620 14:34:33.002259 23334 tablet_service.cc:1514] Processing DeleteTablet for tablet aff0476f94f44779b40aee73d626dd52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:32 UTC) from {username='slave'} at 127.0.0.1:51412
I20260620 14:34:33.002637 23439 tablet_replica.cc:330] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544: stopping tablet replica
I20260620 14:34:33.002669 23441 tablet_replica.cc:330] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: stopping tablet replica
I20260620 14:34:33.002817 23440 tablet_replica.cc:330] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: stopping tablet replica
I20260620 14:34:33.003012 23441 raft_consensus.cc:2229] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:33.003012 23439 raft_consensus.cc:2229] T aff0476f94f44779b40aee73d626dd52 P cce523df3a084215a2ff20377bd20544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:33.003139 23440 raft_consensus.cc:2229] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:33.003160 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23010
I20260620 14:34:33.003582 23440 raft_consensus.cc:2258] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:33.004591 23441 raft_consensus.cc:2258] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:33.005249 23440 ts_tablet_manager.cc:1905] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:33.006304 23441 ts_tablet_manager.cc:1905] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:33.008164 23440 ts_tablet_manager.cc:1918] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:33.008322 23440 log.cc:1196] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/aff0476f94f44779b40aee73d626dd52
I20260620 14:34:33.008630 23440 ts_tablet_manager.cc:1939] T aff0476f94f44779b40aee73d626dd52 P d755b2f30d2b42cbbf6297746216a211: Deleting consensus metadata
I20260620 14:34:33.009120 23441 ts_tablet_manager.cc:1918] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:33.009267 23441 log.cc:1196] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/aff0476f94f44779b40aee73d626dd52
I20260620 14:34:33.009590 23441 ts_tablet_manager.cc:1939] T aff0476f94f44779b40aee73d626dd52 P 26a970e2e63e481089f701124e4e2ee3: Deleting consensus metadata
I20260620 14:34:33.009923 22936 catalog_manager.cc:4841] TS d755b2f30d2b42cbbf6297746216a211 (127.4.58.66:41735): tablet aff0476f94f44779b40aee73d626dd52 (table test-table [id=7d742e60f9be4e68a3d4ddc185a4b322]) successfully deleted
W20260620 14:34:33.010746 22934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:42919 (error 108)
I20260620 14:34:33.010746 22937 catalog_manager.cc:4841] TS 26a970e2e63e481089f701124e4e2ee3 (127.4.58.67:37641): tablet aff0476f94f44779b40aee73d626dd52 (table test-table [id=7d742e60f9be4e68a3d4ddc185a4b322]) successfully deleted
I20260620 14:34:33.011587 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23141
I20260620 14:34:33.011812 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:33.011518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42919 (local address 127.0.0.1:51918)
0620 14:34:33.011657 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:33.011729 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:42919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:34:33.012494 22934 catalog_manager.cc:4584] TS cce523df3a084215a2ff20377bd20544 (127.4.58.65:42919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aff0476f94f44779b40aee73d626dd52: Network error: Client connection negotiation failed: client connection to 127.4.58.65:42919: connect: Connection refused (error 111)
I20260620 14:34:33.016742 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23272
I20260620 14:34:33.021914 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 22919
2026-06-20T14:34:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1120 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:33 dist-test-slave-gqzk krb5kdc[23445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:33 dist-test-slave-gqzk krb5kdc[23445](info): set up 2 sockets
Jun 20 14:34:33 dist-test-slave-gqzk krb5kdc[23445](info): commencing operation
krb5kdc: starting...
W20260620 14:34:33.996628 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:34.115069 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33205
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:33205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:34.223800 23458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:34.224069 23458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:34.224109 23458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:34.228106 23458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:34.228174 23458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:34.228193 23458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:34.228209 23458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:34.228226 23458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:34.233556 23458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:33205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:34.234635 23458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:34.235635 23458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:34.244225 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.244324 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:34.244319 23458 server_base.cc:1034] running on GCE node
W20260620 14:34:34.244243 23466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:34.245064 23458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:34.246140 23458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:34.247303 23458 hybrid_clock.cc:648] HybridClock initialized: now 1781966074247275 us; error 37 us; skew 500 ppm
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:34.249971 23458 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:34.250866 23458 webserver.cc:460] Webserver started at http://127.4.58.126:42883/ using document root <none> and password file <none>
I20260620 14:34:34.251367 23458 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:34.251425 23458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:34.251587 23458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:34.253600 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "9919b3a46e1448b993465d5e92d6f8c9"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f02a63557a96d7143c7319fd9494ab15"
server_key_iv: "dfe99e9c62daa22b51b92f65f81d6b7c"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.254099 23458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "9919b3a46e1448b993465d5e92d6f8c9"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f02a63557a96d7143c7319fd9494ab15"
server_key_iv: "dfe99e9c62daa22b51b92f65f81d6b7c"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.257747 23458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:34.260090 23473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.261307 23458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:34.261411 23458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "9919b3a46e1448b993465d5e92d6f8c9"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f02a63557a96d7143c7319fd9494ab15"
server_key_iv: "dfe99e9c62daa22b51b92f65f81d6b7c"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.261511 23458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:34.281858 23458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:34.284864 23458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:34.285039 23458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:34.291678 23458 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:33205
I20260620 14:34:34.291697 23525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:33205 every 8 connection(s)
I20260620 14:34:34.292622 23458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:34.295415 23526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:34.300976 23526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: Bootstrap starting.
I20260620 14:34:34.301425 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23458
I20260620 14:34:34.301558 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:34.301813 4329 external_mini_cluster.cc:1428] Setting key da00497f50bcfd3e165933d7bebe813f
I20260620 14:34:34.303581 23526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:34.304294 23526 log.cc:824] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:34.306542 23526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: No bootstrap required, opened a new log
I20260620 14:34:34.310899 23526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } }
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:34.311244 23526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:34.311319 23526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9919b3a46e1448b993465d5e92d6f8c9, State: Initialized, Role: FOLLOWER
I20260620 14:34:34.311897 23526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } }
I20260620 14:34:34.312069 23526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:34.312177 23526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:34.312322 23526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:34.313571 23526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } }
I20260620 14:34:34.313972 23526 leader_election.cc:302] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9919b3a46e1448b993465d5e92d6f8c9; no voters:
I20260620 14:34:34.314352 23526 leader_election.cc:288] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:34.314445 23531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:34.314661 23531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 9919b3a46e1448b993465d5e92d6f8c9, State: Running, Role: LEADER
I20260620 14:34:34.315035 23531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } }
I20260620 14:34:34.315752 23526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:34.316884 23533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9919b3a46e1448b993465d5e92d6f8c9. Latest consensus state: current_term: 1 leader_uuid: "9919b3a46e1448b993465d5e92d6f8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } } }
I20260620 14:34:34.317018 23533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:34.317468 23540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:34.317607 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:34.303175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.4.58.126:33205)
0620 14:34:34.303581 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:34.303596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:34.303635 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:34:34.304352 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:34.304381 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:34.304824 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:34.305186 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:34.306045 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.307116 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.308318 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.308859 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.308987 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:34.311881 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:34.311887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:34.311905 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:34.311941 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:34.313969 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.314543 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.314550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.314557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.314766 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.315137 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.315141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.315142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.315613 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:34.315818 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:34.316176 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:34:34.316503 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260620 14:34:34.318351 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9919b3a46e1448b993465d5e92d6f8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919b3a46e1448b993465d5e92d6f8c9" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 33205 } } }
I20260620 14:34:34.318444 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:34.320947 23540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:34.326887 23540 catalog_manager.cc:1347] Generated new cluster ID: 331e30aa086e4bebacf8c7c7f61fedfb
I20260620 14:34:34.326969 23540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:34.340678 23540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:34.342018 23540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:34.359413 23540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: Generated new TSK 0
I20260620 14:34:34.360296 23540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.58.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
Entry for principal kudu/127.4.58.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab.
I20260620 14:34:34.387265 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:34.497284 23552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:34.497547 23552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:34.497591 23552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:34.501595 23552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:34.501663 23552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:34.501729 23552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:34.507009 23552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.65.keytab
--principal=kudu/127.4.58.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:34.508025 23552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:34.509074 23552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:34.519018 23552 server_base.cc:1034] running on GCE node
W20260620 14:34:34.518968 23557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.518965 23558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.518970 23560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:34.519613 23552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:34.520289 23552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:34.521528 23552 hybrid_clock.cc:648] HybridClock initialized: now 1781966074521507 us; error 36 us; skew 500 ppm
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:34.524250 23552 init.cc:377] Logged in from keytab as kudu/127.4.58.65@KRBTEST.COM (short username kudu)
I20260620 14:34:34.525185 23552 webserver.cc:460] Webserver started at http://127.4.58.65:37897/ using document root <none> and password file <none>
I20260620 14:34:34.525748 23552 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:34.525816 23552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:34.526026 23552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:34.528136 23552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "fe94daa6e0384933991370c2b8ce90d1"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "ce0bb274a0618350cd85fa55c71eecbf"
server_key_iv: "a7929020833f890ca5165013fdf6b8fe"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.528740 23552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "fe94daa6e0384933991370c2b8ce90d1"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "ce0bb274a0618350cd85fa55c71eecbf"
server_key_iv: "a7929020833f890ca5165013fdf6b8fe"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.532426 23552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:34:34.535013 23567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.536237 23552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:34.536367 23552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "fe94daa6e0384933991370c2b8ce90d1"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "ce0bb274a0618350cd85fa55c71eecbf"
server_key_iv: "a7929020833f890ca5165013fdf6b8fe"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.536460 23552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:34.559995 23552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:34.563058 23552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:34.563246 23552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:34.563898 23552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:34.565023 23552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:34.565081 23552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.565131 23552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:34.565155 23552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.574029 23552 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:33523
I20260620 14:34:34.574069 23680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:33523 every 8 connection(s)
I20260620 14:34:34.574962 23552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:34.583676 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23552
I20260620 14:34:34.583829 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:34.584110 4329 external_mini_cluster.cc:1428] Setting key e421985e8a4ba97ae7afd07fed34c695
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.65@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:34.588608 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:34.576837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:45511 (local address 127.4.58.126:33205)
0620 14:34:34.576959 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:34.576962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:34.577846 (+ 884us) server_negotiation.cc:407] Connection header received
0620 14:34:34.578758 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:34.578761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:34.578818 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:34.578903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:34.580266 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.580844 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.582012 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.582271 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.582364 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:34.584964 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:34.584967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:34.584969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:34.584997 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:34.586415 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.587019 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.587023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.587025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.587122 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.587453 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.587457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.587458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.587679 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:34.587784 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:34.588309 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:34:34.588445 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:34:34.589541 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:34.577190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33205 (local address 127.4.58.65:45511)
0620 14:34:34.577664 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:34.577709 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:34.578551 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:34.579090 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:34.579097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:34.579544 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:34.580112 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:34.580127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.580971 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.580974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:34.581891 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:34.581902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.582422 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.582427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:34.582607 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:34.583147 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:34.583164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:34.584797 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:34.586536 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:34.586543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:34.586561 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:34.586915 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:34.587216 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:34.587218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:34.587219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:34.587371 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:34.587790 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:34.587798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:34.588062 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 14:34:34.588241 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:34:34.588631 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260620 14:34:34.590999 23681 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33205
I20260620 14:34:34.591240 23681 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:34.591750 23681 heartbeater.cc:511] Master 127.4.58.126:33205 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.58.66@KRBTEST.COM; defaulting to no policy
I20260620 14:34:34.593487 23490 ts_manager.cc:194] Registered new tserver with Master: fe94daa6e0384933991370c2b8ce90d1 (127.4.58.65:33523)
I20260620 14:34:34.594732 23490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.65@KRBTEST.COM'} at 127.4.58.65:45511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
Entry for principal kudu/127.4.58.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab.
I20260620 14:34:34.612493 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:34.721555 23686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:34.721813 23686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:34.721875 23686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:34.725920 23686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:34.725978 23686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:34.726058 23686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:34.731298 23686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.66.keytab
--principal=kudu/127.4.58.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:34.732333 23686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:34.733376 23686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:34.743257 23691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.743278 23692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.743324 23694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:34.743849 23686 server_base.cc:1034] running on GCE node
I20260620 14:34:34.744274 23686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:34.744992 23686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:34.746153 23686 hybrid_clock.cc:648] HybridClock initialized: now 1781966074746128 us; error 38 us; skew 500 ppm
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:34.748930 23686 init.cc:377] Logged in from keytab as kudu/127.4.58.66@KRBTEST.COM (short username kudu)
I20260620 14:34:34.749827 23686 webserver.cc:460] Webserver started at http://127.4.58.66:44485/ using document root <none> and password file <none>
I20260620 14:34:34.750401 23686 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:34.750469 23686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:34.750669 23686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:34.752810 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "d080b6b57c6d420f8a26c58efe205983"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "750f166aea5d97739dfbcd343fb4d17b"
server_key_iv: "279e6ec99962081845ff3c89d10fe618"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.753371 23686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "d080b6b57c6d420f8a26c58efe205983"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "750f166aea5d97739dfbcd343fb4d17b"
server_key_iv: "279e6ec99962081845ff3c89d10fe618"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.757208 23686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:34:34.759629 23701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.760924 23686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:34.761049 23686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "d080b6b57c6d420f8a26c58efe205983"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "750f166aea5d97739dfbcd343fb4d17b"
server_key_iv: "279e6ec99962081845ff3c89d10fe618"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.761159 23686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:34.777819 23686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:34.781224 23686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:34.781402 23686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:34.781993 23686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:34.782810 23686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:34.782856 23686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.782902 23686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:34.782930 23686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.791687 23686 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:36205
I20260620 14:34:34.791710 23814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:36205 every 8 connection(s)
I20260620 14:34:34.792569 23686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:34.799109 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23686
I20260620 14:34:34.799229 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:34.799494 4329 external_mini_cluster.cc:1428] Setting key 5f253c40c077bd59b7d1e71e159efb51
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.66@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:34.806385 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:34.794389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:35519 (local address 127.4.58.126:33205)
0620 14:34:34.794566 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:34.794572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:34.795467 (+ 895us) server_negotiation.cc:407] Connection header received
0620 14:34:34.796388 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:34.796391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:34.796449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:34.796525 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:34.797766 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.798309 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.799491 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.799752 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.799842 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:34.802605 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:34.802610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:34.802613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:34.802645 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:34.804049 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.804680 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.804683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.804684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.804794 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:34.805173 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:34.805176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:34.805178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:34.805391 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:34.805466 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:34.806130 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 14:34:34.806265 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:34:34.807597 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:34.794827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33205 (local address 127.4.58.66:35519)
0620 14:34:34.795305 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:34.795351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:34.796177 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:34.796730 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:34.796738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:34.797135 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:34.797620 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:34.797634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.798428 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.798432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:34.799377 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:34.799387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:34.799910 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:34.799914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:34.800080 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:34.800651 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:34.800670 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:34.802409 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:34.804187 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:34.804194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:34.804212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:34.804554 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:34.804879 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:34.804882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:34.804884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:34.805073 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:34.805483 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:34.805491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:34.805867 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 14:34:34.806097 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 14:34:34.806474 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260620 14:34:34.808849 23815 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33205
I20260620 14:34:34.809082 23815 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.58.67@KRBTEST.COM; defaulting to no policy
I20260620 14:34:34.809587 23815 heartbeater.cc:511] Master 127.4.58.126:33205 requested a full tablet report, sending...
I20260620 14:34:34.810624 23490 ts_manager.cc:194] Registered new tserver with Master: d080b6b57c6d420f8a26c58efe205983 (127.4.58.66:36205)
I20260620 14:34:34.811194 23490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.66@KRBTEST.COM'} at 127.4.58.66:35519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
Entry for principal kudu/127.4.58.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab.
I20260620 14:34:34.825953 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:34.934890 23820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:34.935145 23820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:34.935187 23820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:34.939248 23820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:34.939330 23820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:34.939410 23820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:34.944849 23820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.67.keytab
--principal=kudu/127.4.58.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:33205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:34.945902 23820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:34.946892 23820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:34.956569 23825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.956561 23828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:34.956561 23826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:34.956831 23820 server_base.cc:1034] running on GCE node
I20260620 14:34:34.957291 23820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:34.957926 23820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:34.959081 23820 hybrid_clock.cc:648] HybridClock initialized: now 1781966074959061 us; error 30 us; skew 500 ppm
Jun 20 14:34:34 dist-test-slave-gqzk krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:34.961838 23820 init.cc:377] Logged in from keytab as kudu/127.4.58.67@KRBTEST.COM (short username kudu)
I20260620 14:34:34.962733 23820 webserver.cc:460] Webserver started at http://127.4.58.67:33473/ using document root <none> and password file <none>
I20260620 14:34:34.963274 23820 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:34.963336 23820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:34.963532 23820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:34.965809 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "3fbe2a3ca1fd4938aadc4079fe99a026"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f44d47edb72dc823bc1ae9bce016f5b2"
server_key_iv: "2c0623dc78aa73a8f7c5c141a21ce131"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.966369 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "3fbe2a3ca1fd4938aadc4079fe99a026"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f44d47edb72dc823bc1ae9bce016f5b2"
server_key_iv: "2c0623dc78aa73a8f7c5c141a21ce131"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.970032 23820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:34.972544 23835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:34.973817 23820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:34.973932 23820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "3fbe2a3ca1fd4938aadc4079fe99a026"
format_stamp: "Formatted at 2026-06-20 14:34:34 on dist-test-slave-gqzk"
server_key: "f44d47edb72dc823bc1ae9bce016f5b2"
server_key_iv: "2c0623dc78aa73a8f7c5c141a21ce131"
server_key_version: "encryptionkey@0"
I20260620 14:34:34.974027 23820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:35.016109 23820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:35.019237 23820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:35.019431 23820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:35.020031 23820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:35.020895 23820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:35.020946 23820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.020999 23820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:35.021025 23820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.029526 23820 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:35689
I20260620 14:34:35.029553 23948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:35689 every 8 connection(s)
I20260620 14:34:35.030447 23820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:35.032503 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23820
I20260620 14:34:35.032621 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:35.032920 4329 external_mini_cluster.cc:1428] Setting key de676dc79d07e2099630c396ca3cdf98
Jun 20 14:34:35 dist-test-slave-gqzk krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966074, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.67@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:35.044307 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.032262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:52515 (local address 127.4.58.126:33205)
0620 14:34:35.032424 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.032429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.033443 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 14:34:35.034495 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.034499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.034549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.034623 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:35.035834 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.036380 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.037754 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.038002 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.038085 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.040625 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:35.040628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:35.040630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:35.040656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:35.042108 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:35.042629 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:35.042632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:35.042634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:35.042718 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:35.043153 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:35.043156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:35.043157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:35.043353 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:35.043433 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.044020 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.044179 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:34:35.045244 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.032631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:33205 (local address 127.4.58.67:52515)
0620 14:34:35.033184 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.033341 (+ 157us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.034279 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.034805 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.034816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.035180 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:35.035681 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.035695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.036514 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.036517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.037618 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.037629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.038168 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.038172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.038331 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.038871 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:34:35.038889 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:35.040451 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:35.042220 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:35.042225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:35.042239 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:35.042503 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:35.042830 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:34:35.042833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:34:35.042835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:34:35.042997 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:34:35.043482 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:35.043490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:35.043768 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.043972 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.044307 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":118,"threads_started":1}
I20260620 14:34:35.046339 23949 heartbeater.cc:347] Connected to a master server at 127.4.58.126:33205
I20260620 14:34:35.046577 23949 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:35.047083 23949 heartbeater.cc:511] Master 127.4.58.126:33205 requested a full tablet report, sending...
I20260620 14:34:35.048074 23489 ts_manager.cc:194] Registered new tserver with Master: 3fbe2a3ca1fd4938aadc4079fe99a026 (127.4.58.67:35689)
I20260620 14:34:35.048640 23489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.58.67@KRBTEST.COM'} at 127.4.58.67:52515
I20260620 14:34:35.058881 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:35.068540 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.060883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.4.58.126:33205)
0620 14:34:35.061025 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.061029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.061212 (+ 183us) server_negotiation.cc:407] Connection header received
0620 14:34:35.061313 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.061316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.061363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.061426 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:35.062164 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.062675 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.063498 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.063725 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.063812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.065396 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:35.065399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:35.065401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:35.065431 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:35.067021 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:35.067456 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:35.067459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:35.067460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:35.067548 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:35.067771 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:35.067774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:35.067775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:35.068002 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:35.068123 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.068264 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.068376 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:34:35.071825 23489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:35.074944 23489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:35.089300 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.083401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:33523 (local address 127.0.0.1:55944)
0620 14:34:35.084033 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.084085 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.084204 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.085350 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.085356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.085405 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:35.085534 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.085540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.086505 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.086508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.087731 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.087742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.088713 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.088717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.088856 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.088896 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.089002 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.089131 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":39,"spinlock_wait_cycles":35840,"thread_start_us":85,"threads_started":1}
I20260620 14:34:35.089304 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.083549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.4.58.65:33523)
0620 14:34:35.084881 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.084896 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.084918 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:34:35.084994 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.084999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.085197 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.085348 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:35.085648 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.086388 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.087870 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.088596 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.088694 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.088898 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.089052 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.089147 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1230,"thread_start_us":106,"threads_started":1}
I20260620 14:34:35.089763 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.083772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36205 (local address 127.0.0.1:52162)
0620 14:34:35.084556 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.084568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.084640 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.085350 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.085356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.085398 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:35.085575 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.085581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.087039 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.087042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.088089 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.088097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.089367 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.089371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.089473 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.089495 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.089562 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.089627 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":115,"spinlock_wait_cycles":22272,"thread_start_us":94,"threads_started":1}
I20260620 14:34:35.090046 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.083560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:35689 (local address 127.0.0.1:33720)
0620 14:34:35.084290 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.084304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.084395 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.085350 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.085356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.085400 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:35.085568 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.085575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.087021 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.087024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.088187 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.088197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.089367 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.089371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.089664 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.089685 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.089746 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.089807 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":52,"spinlock_wait_cycles":14208,"thread_start_us":93,"threads_started":1}
I20260620 14:34:35.090457 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.084210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.4.58.67:35689)
0620 14:34:35.084675 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.084686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.084708 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:34:35.084809 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.084814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.085019 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.085170 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:35.085695 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.086874 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.088323 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.089080 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.089176 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.089314 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.090211 (+ 897us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.090313 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":182,"threads_started":1}
I20260620 14:34:35.090577 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.084188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.4.58.66:36205)
0620 14:34:35.084717 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.084728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.084745 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:34:35.084855 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.084859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.085019 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.085170 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:35.085695 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.086880 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.088203 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.089074 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.089176 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.089306 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.089603 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.090435 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":138,"threads_started":1}
I20260620 14:34:35.092466 23615 tablet_service.cc:1467] Processing CreateTablet for tablet 1017652cac054c59a31b384ec0408505 (DEFAULT_TABLE table=test-table [id=ed64ef3165d149e2b006c29a91d96c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:35.093494 23615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017652cac054c59a31b384ec0408505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:35.093323 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 1017652cac054c59a31b384ec0408505 (DEFAULT_TABLE table=test-table [id=ed64ef3165d149e2b006c29a91d96c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:35.093321 23883 tablet_service.cc:1467] Processing CreateTablet for tablet 1017652cac054c59a31b384ec0408505 (DEFAULT_TABLE table=test-table [id=ed64ef3165d149e2b006c29a91d96c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:35.094357 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017652cac054c59a31b384ec0408505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:35.094355 23883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017652cac054c59a31b384ec0408505. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:35.099639 23966 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: Bootstrap starting.
I20260620 14:34:35.101305 23967 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Bootstrap starting.
I20260620 14:34:35.101727 23968 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Bootstrap starting.
I20260620 14:34:35.102532 23966 tablet_bootstrap.cc:654] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:35.103411 23966 log.cc:824] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:35.104344 23968 tablet_bootstrap.cc:654] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:35.105232 23968 log.cc:824] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:35.105351 23966 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: No bootstrap required, opened a new log
I20260620 14:34:35.105545 23966 ts_tablet_manager.cc:1397] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:35.106287 23967 tablet_bootstrap.cc:654] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:35.107086 23967 log.cc:824] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:35.107270 23968 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: No bootstrap required, opened a new log
I20260620 14:34:35.107460 23968 ts_tablet_manager.cc:1397] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:34:35.109086 23967 tablet_bootstrap.cc:492] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: No bootstrap required, opened a new log
I20260620 14:34:35.109282 23967 ts_tablet_manager.cc:1397] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260620 14:34:35.109530 23966 raft_consensus.cc:348] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.109748 23966 raft_consensus.cc:374] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:35.109818 23966 raft_consensus.cc:729] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe94daa6e0384933991370c2b8ce90d1, State: Initialized, Role: FOLLOWER
I20260620 14:34:35.110251 23966 consensus_queue.cc:260] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.110893 23681 heartbeater.cc:503] Master 127.4.58.126:33205 was elected leader, sending a full tablet report...
I20260620 14:34:35.111222 23966 ts_tablet_manager.cc:1428] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:35.111343 23968 raft_consensus.cc:348] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.111608 23968 raft_consensus.cc:374] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:35.111650 23968 raft_consensus.cc:729] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d080b6b57c6d420f8a26c58efe205983, State: Initialized, Role: FOLLOWER
I20260620 14:34:35.112126 23968 consensus_queue.cc:260] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.113132 23968 ts_tablet_manager.cc:1428] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:35.112996 23967 raft_consensus.cc:348] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.113155 23815 heartbeater.cc:503] Master 127.4.58.126:33205 was elected leader, sending a full tablet report...
I20260620 14:34:35.113296 23967 raft_consensus.cc:374] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:35.113358 23967 raft_consensus.cc:729] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbe2a3ca1fd4938aadc4079fe99a026, State: Initialized, Role: FOLLOWER
I20260620 14:34:35.113827 23967 consensus_queue.cc:260] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.114742 23967 ts_tablet_manager.cc:1428] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:34:35.115087 23949 heartbeater.cc:503] Master 127.4.58.126:33205 was elected leader, sending a full tablet report...
I20260620 14:34:35.214705 23974 raft_consensus.cc:482] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:35.214923 23974 raft_consensus.cc:504] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.216099 23974 leader_election.cc:288] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe94daa6e0384933991370c2b8ce90d1 (127.4.58.65:33523), d080b6b57c6d420f8a26c58efe205983 (127.4.58.66:36205)
I20260620 14:34:35.220319 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.216489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:39787 (local address 127.4.58.65:33523)
0620 14:34:35.216600 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.216605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.216623 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:34:35.216862 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.216867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.216924 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.216998 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:35.217267 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.217944 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.219203 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.219726 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.219807 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.219863 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.220094 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.220174 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:34:35.220346 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.216378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:33523 (local address 127.4.58.67:39787)
0620 14:34:35.216568 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.216582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.216677 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.217016 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.217019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.217037 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:35.217160 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.217166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.218049 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.218053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.219057 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.219067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.219850 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.219855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.219966 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.220002 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.220070 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.220212 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1}
I20260620 14:34:35.220476 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.216495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:47687 (local address 127.4.58.66:36205)
0620 14:34:35.216615 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.216619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.216950 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:34:35.217110 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.217113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.217164 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.217243 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:35.217560 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.218267 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.219483 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.219949 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.220051 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.220109 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.220303 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.220374 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:34:35.220666 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.216380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:36205 (local address 127.4.58.67:47687)
0620 14:34:35.216858 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.216870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.216967 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.217267 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.217270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.217308 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:35.217424 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.217431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.218412 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.218415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.219354 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.219364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.220066 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.220070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.220167 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.220199 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.220271 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.220330 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":39,"thread_start_us":133,"threads_started":1}
I20260620 14:34:35.221118 23635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017652cac054c59a31b384ec0408505" candidate_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe94daa6e0384933991370c2b8ce90d1" is_pre_election: true
I20260620 14:34:35.221271 23769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017652cac054c59a31b384ec0408505" candidate_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d080b6b57c6d420f8a26c58efe205983" is_pre_election: true
I20260620 14:34:35.221451 23635 raft_consensus.cc:2454] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbe2a3ca1fd4938aadc4079fe99a026 in term 0.
I20260620 14:34:35.221549 23769 raft_consensus.cc:2454] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbe2a3ca1fd4938aadc4079fe99a026 in term 0.
I20260620 14:34:35.222005 23836 leader_election.cc:302] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fbe2a3ca1fd4938aadc4079fe99a026, d080b6b57c6d420f8a26c58efe205983, fe94daa6e0384933991370c2b8ce90d1; no voters:
I20260620 14:34:35.222275 23974 raft_consensus.cc:2781] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:35.222357 23974 raft_consensus.cc:482] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:35.222404 23974 raft_consensus.cc:3037] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:35.223390 23974 raft_consensus.cc:504] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.223734 23974 leader_election.cc:288] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [CANDIDATE]: Term 1 election: Requested vote from peers fe94daa6e0384933991370c2b8ce90d1 (127.4.58.65:33523), d080b6b57c6d420f8a26c58efe205983 (127.4.58.66:36205)
I20260620 14:34:35.224095 23769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017652cac054c59a31b384ec0408505" candidate_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d080b6b57c6d420f8a26c58efe205983"
I20260620 14:34:35.224103 23635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017652cac054c59a31b384ec0408505" candidate_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe94daa6e0384933991370c2b8ce90d1"
I20260620 14:34:35.224213 23769 raft_consensus.cc:3037] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:35.224215 23635 raft_consensus.cc:3037] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:35.225250 23769 raft_consensus.cc:2454] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbe2a3ca1fd4938aadc4079fe99a026 in term 1.
I20260620 14:34:35.225287 23635 raft_consensus.cc:2454] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbe2a3ca1fd4938aadc4079fe99a026 in term 1.
I20260620 14:34:35.225528 23836 leader_election.cc:302] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fbe2a3ca1fd4938aadc4079fe99a026, d080b6b57c6d420f8a26c58efe205983; no voters:
I20260620 14:34:35.225679 23974 raft_consensus.cc:2781] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:35.225879 23974 raft_consensus.cc:686] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbe2a3ca1fd4938aadc4079fe99a026, State: Running, Role: LEADER
I20260620 14:34:35.226181 23974 consensus_queue.cc:237] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } }
I20260620 14:34:35.229410 23489 catalog_manager.cc:5495] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fbe2a3ca1fd4938aadc4079fe99a026 (127.4.58.67). New cstate: current_term: 1 leader_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fbe2a3ca1fd4938aadc4079fe99a026" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 35689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 } health_report { overall_health: UNKNOWN } } }
I20260620 14:34:35.243753 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.240308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.4.58.67:35689)
0620 14:34:35.240439 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.240442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.240455 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:34:35.240544 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.240546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.240601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.240707 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:35.240962 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.241509 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.242383 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.242591 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.242673 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.242958 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:35.243367 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:35.243471 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.243537 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.243594 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:34:35.249415 23769 raft_consensus.cc:1261] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 1 FOLLOWER]: Refusing update from remote peer 3fbe2a3ca1fd4938aadc4079fe99a026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:35.249431 23635 raft_consensus.cc:1261] T 1017652cac054c59a31b384ec0408505 P fe94daa6e0384933991370c2b8ce90d1 [term 1 FOLLOWER]: Refusing update from remote peer 3fbe2a3ca1fd4938aadc4079fe99a026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:35.250072 23974 consensus_queue.cc:1035] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe94daa6e0384933991370c2b8ce90d1" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:35.250265 23976 consensus_queue.cc:1035] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d080b6b57c6d420f8a26c58efe205983" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:35.256079 23979 mvcc.cc:205] Tried to move back new op lower bound from 7298933044214788096 to 7298933044128923648. Current Snapshot: MvccSnapshot[applied={T|T < 7298933044214788096}]
I20260620 14:34:35.257774 23981 mvcc.cc:205] Tried to move back new op lower bound from 7298933044214788096 to 7298933044128923648. Current Snapshot: MvccSnapshot[applied={T|T < 7298933044214788096}]
I20260620 14:34:35.260691 23980 mvcc.cc:205] Tried to move back new op lower bound from 7298933044214788096 to 7298933044128923648. Current Snapshot: MvccSnapshot[applied={T|T < 7298933044214788096 or (T in {7298933044214788096})}]
I20260620 14:34:35.266170 23489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:35.266395 23489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:35.268640 23489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9919b3a46e1448b993465d5e92d6f8c9: Sending DeleteTablet for 3 replicas of tablet 1017652cac054c59a31b384ec0408505
I20260620 14:34:35.269434 23615 tablet_service.cc:1514] Processing DeleteTablet for tablet 1017652cac054c59a31b384ec0408505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:35 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:55944
I20260620 14:34:35.269544 23749 tablet_service.cc:1514] Processing DeleteTablet for tablet 1017652cac054c59a31b384ec0408505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:35 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:52162
I20260620 14:34:35.269996 23883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1017652cac054c59a31b384ec0408505 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:35 UTC) from {username='kudu', principal='kudu/127.4.58.126@KRBTEST.COM'} at 127.0.0.1:33720
I20260620 14:34:35.270267 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23552
I20260620 14:34:35.270422 23990 tablet_replica.cc:330] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: stopping tablet replica
I20260620 14:34:35.270826 23990 raft_consensus.cc:2229] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:35.270991 23989 tablet_replica.cc:330] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: stopping tablet replica
I20260620 14:34:35.271274 23990 raft_consensus.cc:2258] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:35.271345 23989 raft_consensus.cc:2229] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:35.271692 23989 raft_consensus.cc:2258] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:35.273169 23989 ts_tablet_manager.cc:1905] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:35.273233 23990 ts_tablet_manager.cc:1905] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:35.275890 23989 ts_tablet_manager.cc:1918] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:35.275890 23990 ts_tablet_manager.cc:1918] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:35.275990 23990 log.cc:1196] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/1017652cac054c59a31b384ec0408505
I20260620 14:34:35.276101 23989 log.cc:1196] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/1017652cac054c59a31b384ec0408505
I20260620 14:34:35.276321 23990 ts_tablet_manager.cc:1939] T 1017652cac054c59a31b384ec0408505 P 3fbe2a3ca1fd4938aadc4079fe99a026: Deleting consensus metadata
I20260620 14:34:35.276458 23989 ts_tablet_manager.cc:1939] T 1017652cac054c59a31b384ec0408505 P d080b6b57c6d420f8a26c58efe205983: Deleting consensus metadata
I20260620 14:34:35.277690 23474 catalog_manager.cc:4841] TS d080b6b57c6d420f8a26c58efe205983 (127.4.58.66:36205): tablet 1017652cac054c59a31b384ec0408505 (table test-table [id=ed64ef3165d149e2b006c29a91d96c9f]) successfully deleted
I20260620 14:34:35.277676 23477 catalog_manager.cc:4841] TS 3fbe2a3ca1fd4938aadc4079fe99a026 (127.4.58.67:35689): tablet 1017652cac054c59a31b384ec0408505 (table test-table [id=ed64ef3165d149e2b006c29a91d96c9f]) successfully deleted
W20260620 14:34:35.280014 23475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:33523 (error 108)
I20260620 14:34:35.280830 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23686
I20260620 14:34:35.281023 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.280730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:33523 (local address 127.0.0.1:55950)
0620 14:34:35.280871 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.280941 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:33523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:34:35.281471 23475 catalog_manager.cc:4584] TS fe94daa6e0384933991370c2b8ce90d1 (127.4.58.65:33523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1017652cac054c59a31b384ec0408505: Network error: Client connection negotiation failed: client connection to 127.4.58.65:33523: connect: Connection refused (error 111)
I20260620 14:34:35.289112 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23820
I20260620 14:34:35.294872 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23458
2026-06-20T14:34:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2275 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:35Z Disabled control of system clock
I20260620 14:34:35.325217 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34591
--webserver_interface=127.4.58.126
--webserver_port=0
--builtin_ntp_servers=127.4.58.84:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:35.435853 23995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:35.436128 23995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:35.436163 23995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:35.440254 23995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:35.440325 23995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:35.440338 23995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:35.440354 23995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:35.440368 23995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:35.445276 23995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:35.446399 23995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:35.447782 23995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:35.456161 24000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:35.456202 24001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:35.456414 23995 server_base.cc:1034] running on GCE node
W20260620 14:34:35.456229 24003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:35.457149 23995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:35.458334 23995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:35.459506 23995 hybrid_clock.cc:648] HybridClock initialized: now 1781966075459490 us; error 31 us; skew 500 ppm
I20260620 14:34:35.461361 23995 webserver.cc:460] Webserver started at http://127.4.58.126:44195/ using document root <none> and password file <none>
I20260620 14:34:35.461953 23995 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:35.462024 23995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:35.462210 23995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:35.464259 23995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "456441b7c2b14710a2baa3ddaae94996"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "768d7f7c9475425b968ce501cd35620c"
server_key_iv: "f3a79359a2310c716fbe8f80eafb2f8e"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.464828 23995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "456441b7c2b14710a2baa3ddaae94996"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "768d7f7c9475425b968ce501cd35620c"
server_key_iv: "f3a79359a2310c716fbe8f80eafb2f8e"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.468565 23995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:34:35.471017 24009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.472266 23995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:35.472389 23995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "456441b7c2b14710a2baa3ddaae94996"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "768d7f7c9475425b968ce501cd35620c"
server_key_iv: "f3a79359a2310c716fbe8f80eafb2f8e"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.472486 23995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:35.481988 23995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:35.482631 23995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:35.482802 23995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:35.489140 23995 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34591
I20260620 14:34:35.489147 24061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34591 every 8 connection(s)
I20260620 14:34:35.490053 23995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:35.490537 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 23995
I20260620 14:34:35.490690 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:35.491008 4329 external_mini_cluster.cc:1428] Setting key 5ca75556be5f6871bca6cf2be71f4826
I20260620 14:34:35.494442 24062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:35.500029 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.492541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.4.58.126:34591)
0620 14:34:35.493145 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.493159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.493196 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:34:35.493833 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.493850 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.494222 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.494549 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:35.494913 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.495880 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.496812 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.497277 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.497435 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.498346 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:35.498351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:35.498364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:35.498733 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:35.498788 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:35.498810 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:35.498891 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.499025 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.499271 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":153,"threads_started":1}
I20260620 14:34:35.500525 24062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: Bootstrap starting.
I20260620 14:34:35.502892 24062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:35.503590 24062 log.cc:824] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:35.505512 24062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: No bootstrap required, opened a new log
I20260620 14:34:35.508855 24062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } }
I20260620 14:34:35.509078 24062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:35.509140 24062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456441b7c2b14710a2baa3ddaae94996, State: Initialized, Role: FOLLOWER
I20260620 14:34:35.509534 24062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } }
I20260620 14:34:35.509665 24062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:35.509727 24062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:35.509819 24062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:35.510661 24062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } }
I20260620 14:34:35.510972 24062 leader_election.cc:302] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456441b7c2b14710a2baa3ddaae94996; no voters:
I20260620 14:34:35.511252 24062 leader_election.cc:288] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:35.511360 24067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:35.511559 24067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [term 1 LEADER]: Becoming Leader. State: Replica: 456441b7c2b14710a2baa3ddaae94996, State: Running, Role: LEADER
I20260620 14:34:35.511850 24067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } }
I20260620 14:34:35.512364 24062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:35.513391 24069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456441b7c2b14710a2baa3ddaae94996. Latest consensus state: current_term: 1 leader_uuid: "456441b7c2b14710a2baa3ddaae94996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } } }
I20260620 14:34:35.513525 24069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:35.513401 24068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456441b7c2b14710a2baa3ddaae94996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456441b7c2b14710a2baa3ddaae94996" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34591 } } }
I20260620 14:34:35.513623 24068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:35.513903 24076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:35.517369 24076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:35.523033 24076 catalog_manager.cc:1347] Generated new cluster ID: e0aca7b8b57949c6bca70ebf9a878924
I20260620 14:34:35.523106 24076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:35.547780 24076 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:35.548803 24076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:35.557106 24076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: Generated new TSK 0
I20260620 14:34:35.557794 24076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:35.564348 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.65:0
--local_ip_for_outbound_sockets=127.4.58.65
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--builtin_ntp_servers=127.4.58.84:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:34:35.685314 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:35.685585 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:35.685639 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:35.689667 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:35.689726 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:35.689803 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.65
I20260620 14:34:35.694737 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.58.65
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.65
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:35.695829 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:35.697325 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:35.706756 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:35.706759 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:35.706827 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:35.706821 24086 server_base.cc:1034] running on GCE node
I20260620 14:34:35.707410 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:35.708019 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:35.709158 24086 hybrid_clock.cc:648] HybridClock initialized: now 1781966075709128 us; error 43 us; skew 500 ppm
I20260620 14:34:35.711030 24086 webserver.cc:460] Webserver started at http://127.4.58.65:39605/ using document root <none> and password file <none>
I20260620 14:34:35.711577 24086 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:35.711637 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:35.711828 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:35.713948 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data/instance:
uuid: "493a182baebd4acf9199008ae7d8659e"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "154b21a1a47249734c83ad4b434f95d1"
server_key_iv: "4dce202ef7f75d9d0b2e8ea2efe7f7ad"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.714483 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance:
uuid: "493a182baebd4acf9199008ae7d8659e"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "154b21a1a47249734c83ad4b434f95d1"
server_key_iv: "4dce202ef7f75d9d0b2e8ea2efe7f7ad"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.718146 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:35.720537 24100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.721799 24086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:35.721904 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal
uuid: "493a182baebd4acf9199008ae7d8659e"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "154b21a1a47249734c83ad4b434f95d1"
server_key_iv: "4dce202ef7f75d9d0b2e8ea2efe7f7ad"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.722009 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:35.739224 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:35.739984 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:35.740165 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:35.740828 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:35.741827 24086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:35.741873 24086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.741914 24086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:35.741942 24086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.750162 24086 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.65:42933
I20260620 14:34:35.750185 24213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.65:42933 every 8 connection(s)
I20260620 14:34:35.751169 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/data/info.pb
I20260620 14:34:35.760376 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.752990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.65:35515 (local address 127.4.58.126:34591)
0620 14:34:35.753127 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.753131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.754308 (+ 1177us) server_negotiation.cc:407] Connection header received
0620 14:34:35.755616 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.755619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.755672 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.755753 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:35.756713 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.757355 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.758529 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.758782 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.758876 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.759894 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:35.759898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:35.759900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:35.759914 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:35.759934 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:35.759942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:35.760000 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.760181 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.760257 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:34:35.761310 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.753470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34591 (local address 127.4.58.65:35515)
0620 14:34:35.754142 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.754184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.755428 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.755929 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.755935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.756026 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:35.756536 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.756550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.757482 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.757485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.758397 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.758406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.758914 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.758917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.759058 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.759612 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:35.759613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:35.759656 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:35.759734 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:35.760007 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:35.760013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:35.760017 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.760108 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.760341 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260620 14:34:35.761404 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24086
I20260620 14:34:35.761520 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-0/wal/instance
I20260620 14:34:35.761873 4329 external_mini_cluster.cc:1428] Setting key 3f610b8b8e58635966a987616965bffb
I20260620 14:34:35.762669 24214 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34591
I20260620 14:34:35.762933 24214 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:35.763576 24214 heartbeater.cc:511] Master 127.4.58.126:34591 requested a full tablet report, sending...
I20260620 14:34:35.763952 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.66:0
--local_ip_for_outbound_sockets=127.4.58.66
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--builtin_ntp_servers=127.4.58.84:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:35.765202 24026 ts_manager.cc:194] Registered new tserver with Master: 493a182baebd4acf9199008ae7d8659e (127.4.58.65:42933)
I20260620 14:34:35.766615 24026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.65:35515
W20260620 14:34:35.873517 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:35.873804 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:35.873863 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:35.878041 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:35.878108 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:35.878185 24217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.66
I20260620 14:34:35.883121 24217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.58.66
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.66
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:35.884231 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:35.885651 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:35.895337 24217 server_base.cc:1034] running on GCE node
W20260620 14:34:35.895275 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:35.895372 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:35.895274 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:35.895910 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:35.896540 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:35.897691 24217 hybrid_clock.cc:648] HybridClock initialized: now 1781966075897658 us; error 51 us; skew 500 ppm
I20260620 14:34:35.899602 24217 webserver.cc:460] Webserver started at http://127.4.58.66:34011/ using document root <none> and password file <none>
I20260620 14:34:35.900161 24217 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:35.900221 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:35.900424 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:35.902509 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data/instance:
uuid: "590c4e463e6247afb6a861e92b92f869"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "5d36e832d6d0044ebd18f56cdf791db8"
server_key_iv: "8afc398ce7a7e0cbfa700c9b3596b578"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.903064 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance:
uuid: "590c4e463e6247afb6a861e92b92f869"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "5d36e832d6d0044ebd18f56cdf791db8"
server_key_iv: "8afc398ce7a7e0cbfa700c9b3596b578"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.906658 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:34:35.909026 24231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.910250 24217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:35.910382 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal
uuid: "590c4e463e6247afb6a861e92b92f869"
format_stamp: "Formatted at 2026-06-20 14:34:35 on dist-test-slave-gqzk"
server_key: "5d36e832d6d0044ebd18f56cdf791db8"
server_key_iv: "8afc398ce7a7e0cbfa700c9b3596b578"
server_key_version: "encryptionkey@0"
I20260620 14:34:35.910475 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:35.937547 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:35.938295 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:35.938476 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:35.939070 24217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:35.940056 24217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:35.940104 24217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.940156 24217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:35.940182 24217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:35.948472 24217 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.66:37069
I20260620 14:34:35.948496 24344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.66:37069 every 8 connection(s)
I20260620 14:34:35.949489 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/data/info.pb
I20260620 14:34:35.950235 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24217
I20260620 14:34:35.950362 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal/instance
I20260620 14:34:35.950670 4329 external_mini_cluster.cc:1428] Setting key 771cc218fcfa2e649732df46f5533792
I20260620 14:34:35.952648 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.58.67:0
--local_ip_for_outbound_sockets=127.4.58.67
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--builtin_ntp_servers=127.4.58.84:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:34:35.959674 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.951368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.66:56845 (local address 127.4.58.126:34591)
0620 14:34:35.951497 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:35.951501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:35.953453 (+ 1952us) server_negotiation.cc:407] Connection header received
0620 14:34:35.954866 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:35.954869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:35.954927 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:35.955003 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:35.956028 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.956555 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.957751 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.957980 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.958072 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.959183 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:35.959186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:35.959188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:35.959202 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:35.959231 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:35.959239 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:35.959304 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:35.959447 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:34:35.959522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:34:35.960546 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:35.951887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34591 (local address 127.4.58.66:56845)
0620 14:34:35.953296 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:35.953340 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:35.954640 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:35.955195 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:35.955203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:35.955294 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:35.955889 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.955905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.956681 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.956684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.957634 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:35.957643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:35.958147 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:35.958152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:35.958314 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:35.958913 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:35.958914 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:35.958960 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:35.959030 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:35.959316 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:35.959324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:35.959331 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:34:35.959442 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:34:35.959760 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":108,"threads_started":1}
I20260620 14:34:35.961880 24345 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34591
I20260620 14:34:35.962127 24345 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:35.962694 24345 heartbeater.cc:511] Master 127.4.58.126:34591 requested a full tablet report, sending...
I20260620 14:34:35.963733 24026 ts_manager.cc:194] Registered new tserver with Master: 590c4e463e6247afb6a861e92b92f869 (127.4.58.66:37069)
I20260620 14:34:35.964363 24026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.66:56845
W20260620 14:34:36.065487 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:36.065764 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:36.065816 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:36.069977 24348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:36.070051 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:36.070147 24348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.58.67
I20260620 14:34:36.075443 24348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.58.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.58.67
--webserver_port=0
--tserver_master_addrs=127.4.58.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.58.67
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:36.076612 24348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:36.078133 24348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:36.087919 24354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:36.087919 24356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:36.087957 24353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:36.088171 24348 server_base.cc:1034] running on GCE node
I20260620 14:34:36.088668 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:36.089339 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:36.090497 24348 hybrid_clock.cc:648] HybridClock initialized: now 1781966076090475 us; error 33 us; skew 500 ppm
I20260620 14:34:36.092377 24348 webserver.cc:460] Webserver started at http://127.4.58.67:33157/ using document root <none> and password file <none>
I20260620 14:34:36.092965 24348 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:36.093029 24348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:36.093191 24348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:36.095229 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data/instance:
uuid: "80c82b4c075d4a32b76cfc747375d0ca"
format_stamp: "Formatted at 2026-06-20 14:34:36 on dist-test-slave-gqzk"
server_key: "e2d0727a8e44d89d4d80330089e74982"
server_key_iv: "3b9a67d30cb7f3855117b71cdbae905a"
server_key_version: "encryptionkey@0"
I20260620 14:34:36.095749 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance:
uuid: "80c82b4c075d4a32b76cfc747375d0ca"
format_stamp: "Formatted at 2026-06-20 14:34:36 on dist-test-slave-gqzk"
server_key: "e2d0727a8e44d89d4d80330089e74982"
server_key_iv: "3b9a67d30cb7f3855117b71cdbae905a"
server_key_version: "encryptionkey@0"
I20260620 14:34:36.099733 24348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:34:36.102241 24362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:36.103571 24348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:36.103672 24348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal
uuid: "80c82b4c075d4a32b76cfc747375d0ca"
format_stamp: "Formatted at 2026-06-20 14:34:36 on dist-test-slave-gqzk"
server_key: "e2d0727a8e44d89d4d80330089e74982"
server_key_iv: "3b9a67d30cb7f3855117b71cdbae905a"
server_key_version: "encryptionkey@0"
I20260620 14:34:36.103767 24348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:36.117666 24348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:36.118412 24348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:36.118572 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:36.119118 24348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:34:36.120060 24348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:34:36.120108 24348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:36.120147 24348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:34:36.120163 24348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:36.128894 24348 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.67:42219
I20260620 14:34:36.128937 24475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.67:42219 every 8 connection(s)
I20260620 14:34:36.129768 24348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/data/info.pb
I20260620 14:34:36.138293 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.131534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:59539 (local address 127.4.58.126:34591)
0620 14:34:36.131701 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.131706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.132743 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 14:34:36.134069 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.134072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.134123 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.134199 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:36.135058 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.135581 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.136643 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.136901 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.136978 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.137927 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:36.137929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:36.137931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:36.137944 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:36.137963 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:36.137971 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:36.138023 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.138120 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.138170 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:34:36.138885 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24348
I20260620 14:34:36.138976 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal/instance
I20260620 14:34:36.139151 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.131959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:34591 (local address 127.4.58.67:59539)
0620 14:34:36.132586 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.132626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.133861 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.134368 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.134375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.134461 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:34:36.134944 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.134956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.135670 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.135672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.136552 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.136560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.137009 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.137012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.137148 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.137679 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:34:36.137681 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:34:36.137719 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:34:36.137783 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:34:36.138024 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:34:36.138028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:34:36.138032 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.138114 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.138345 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260620 14:34:36.139250 4329 external_mini_cluster.cc:1428] Setting key c8fa5850a46ef2b767aa192aa3cd63a8
I20260620 14:34:36.140327 24476 heartbeater.cc:347] Connected to a master server at 127.4.58.126:34591
I20260620 14:34:36.140551 24476 heartbeater.cc:464] Registering TS with master...
I20260620 14:34:36.141166 24476 heartbeater.cc:511] Master 127.4.58.126:34591 requested a full tablet report, sending...
I20260620 14:34:36.142095 24025 ts_manager.cc:194] Registered new tserver with Master: 80c82b4c075d4a32b76cfc747375d0ca (127.4.58.67:42219)
I20260620 14:34:36.142679 24025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.58.67:59539
I20260620 14:34:36.153438 4329 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:34:36.159159 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.155376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.4.58.126:34591)
0620 14:34:36.155515 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.155519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.155727 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:34:36.155849 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.155852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.155900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.155970 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:36.156244 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.156822 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.157701 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.157942 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.158030 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.158754 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:36.158758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:36.158760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:36.158773 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:36.158795 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:36.158803 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:36.158859 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.158954 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.159010 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:34:36.166803 24025 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:34:36.169818 24025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:34:36.183275 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.177963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:37069 (local address 127.0.0.1:57306)
0620 14:34:36.178332 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.178386 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.178512 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.179122 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.179180 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.179216 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:36.179359 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.179367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.180736 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.180740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.181835 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.181845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.182816 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.182820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.182930 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.182977 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.183059 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.183147 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":25,"thread_start_us":72,"threads_started":1}
I20260620 14:34:36.183986 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.178150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.4.58.66:37069)
0620 14:34:36.178485 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.178498 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.178523 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:34:36.178766 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.178772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.178953 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.179099 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:36.179471 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.180611 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.181949 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.182691 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.183463 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.183667 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.183723 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.183819 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
I20260620 14:34:36.184044 24490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.178803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.67:42219 (local address 127.0.0.1:53804)
0620 14:34:36.179242 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.179254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.179349 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.179787 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.179790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.179808 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:36.179917 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.179923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.181225 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.181229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.182347 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.182356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.183678 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.183682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.183776 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.183800 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.183846 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.183907 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":103,"threads_started":1}
I20260620 14:34:36.184878 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.179647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42933 (local address 127.0.0.1:37732)
0620 14:34:36.180141 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.180154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.180297 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.180680 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.180683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.180700 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:36.180880 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.180886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.182204 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.182207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.183429 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.183440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.184488 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.184492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.184585 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.184606 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.184675 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.184736 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":40,"thread_start_us":73,"threads_started":1}
I20260620 14:34:36.184963 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.179300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.4.58.65:42933)
0620 14:34:36.179617 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.179628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.180232 (+ 604us) server_negotiation.cc:407] Connection header received
0620 14:34:36.180425 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.180431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.180562 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.180690 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:36.180975 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.182083 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.183547 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.184297 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.184368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.184537 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.184708 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.184821 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260620 14:34:36.185194 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.178808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.4.58.67:42219)
0620 14:34:36.179108 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.179119 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.179345 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:34:36.179453 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.179458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.179647 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.179807 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:36.180019 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.181105 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.182473 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.183292 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.184464 (+ 1172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.184617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.184659 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.184779 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260620 14:34:36.187354 24148 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b (DEFAULT_TABLE table=test-table [id=b6dae68a5c374f5b9376c3d4f67014ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:36.187374 24279 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b (DEFAULT_TABLE table=test-table [id=b6dae68a5c374f5b9376c3d4f67014ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:36.187541 24410 tablet_service.cc:1467] Processing CreateTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b (DEFAULT_TABLE table=test-table [id=b6dae68a5c374f5b9376c3d4f67014ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:34:36.188300 24148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1f0def91464d5c95c6e59be05e2c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:36.188397 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1f0def91464d5c95c6e59be05e2c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:36.188397 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d1f0def91464d5c95c6e59be05e2c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:36.195184 24493 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Bootstrap starting.
I20260620 14:34:36.196281 24495 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Bootstrap starting.
I20260620 14:34:36.196322 24494 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Bootstrap starting.
I20260620 14:34:36.197659 24493 tablet_bootstrap.cc:654] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:36.198364 24495 tablet_bootstrap.cc:654] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:36.198386 24494 tablet_bootstrap.cc:654] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:36.198560 24493 log.cc:824] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:36.199116 24494 log.cc:824] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:36.199082 24495 log.cc:824] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:36.200968 24493 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: No bootstrap required, opened a new log
I20260620 14:34:36.201150 24494 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: No bootstrap required, opened a new log
I20260620 14:34:36.201150 24495 tablet_bootstrap.cc:492] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: No bootstrap required, opened a new log
I20260620 14:34:36.201159 24493 ts_tablet_manager.cc:1397] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:34:36.201334 24495 ts_tablet_manager.cc:1397] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:34:36.201334 24494 ts_tablet_manager.cc:1397] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:34:36.204348 24493 raft_consensus.cc:348] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.204589 24493 raft_consensus.cc:374] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:36.204641 24493 raft_consensus.cc:729] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493a182baebd4acf9199008ae7d8659e, State: Initialized, Role: FOLLOWER
I20260620 14:34:36.205132 24493 consensus_queue.cc:260] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.205490 24494 raft_consensus.cc:348] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.205765 24494 raft_consensus.cc:374] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:36.205787 24214 heartbeater.cc:503] Master 127.4.58.126:34591 was elected leader, sending a full tablet report...
I20260620 14:34:36.205824 24494 raft_consensus.cc:729] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590c4e463e6247afb6a861e92b92f869, State: Initialized, Role: FOLLOWER
I20260620 14:34:36.206163 24493 ts_tablet_manager.cc:1428] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:34:36.206238 24494 consensus_queue.cc:260] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.205490 24495 raft_consensus.cc:348] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.206389 24495 raft_consensus.cc:374] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:36.206451 24495 raft_consensus.cc:729] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c82b4c075d4a32b76cfc747375d0ca, State: Initialized, Role: FOLLOWER
I20260620 14:34:36.206849 24495 consensus_queue.cc:260] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.207283 24494 ts_tablet_manager.cc:1428] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:34:36.207592 24345 heartbeater.cc:503] Master 127.4.58.126:34591 was elected leader, sending a full tablet report...
I20260620 14:34:36.207803 24495 ts_tablet_manager.cc:1428] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:34:36.208274 24476 heartbeater.cc:503] Master 127.4.58.126:34591 was elected leader, sending a full tablet report...
W20260620 14:34:36.252399 24215 tablet.cc:2369] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:34:36.380697 24477 tablet.cc:2369] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:36.393808 24501 raft_consensus.cc:482] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:34:36.394012 24501 raft_consensus.cc:504] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.395179 24501 leader_election.cc:288] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 590c4e463e6247afb6a861e92b92f869 (127.4.58.66:37069), 493a182baebd4acf9199008ae7d8659e (127.4.58.65:42933)
I20260620 14:34:36.399387 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.395371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:37069 (local address 127.4.58.67:59307)
0620 14:34:36.395651 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.395668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.395794 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.396111 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.396115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.396137 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:36.396263 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.396269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.397121 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.397125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.398160 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.398170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.398925 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.398929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.399033 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.399060 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.399141 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.399217 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 14:34:36.399463 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.395484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:59307 (local address 127.4.58.66:37069)
0620 14:34:36.395628 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.395633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.395765 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:34:36.395932 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.395935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.395988 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.396068 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:36.396403 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.397018 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.398284 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.398800 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.398878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.398931 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.399251 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.399335 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:34:36.399792 24491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.395815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.58.67:48361 (local address 127.4.58.65:42933)
0620 14:34:36.395950 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.395954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.396058 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:34:36.396197 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.396200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.396248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.396331 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:34:36.396684 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.397366 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.398653 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.399169 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.399259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.399317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.399563 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.399645 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:34:36.400154 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.395661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42933 (local address 127.4.58.67:48361)
0620 14:34:36.395977 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.395990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:34:36.396084 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:34:36.396364 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:34:36.396367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:34:36.396386 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:34:36.396516 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.396523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.397497 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.397500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.398520 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:34:36.398530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.399300 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.399304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:34:36.399403 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.399427 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:34:36.399939 (+ 512us) client_negotiation.cc:240] Negotiation successful
0620 14:34:36.400016 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":134,"threads_started":1}
I20260620 14:34:36.400107 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1f0def91464d5c95c6e59be05e2c1b" candidate_uuid: "80c82b4c075d4a32b76cfc747375d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590c4e463e6247afb6a861e92b92f869" is_pre_election: true
I20260620 14:34:36.400429 24299 raft_consensus.cc:2454] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c82b4c075d4a32b76cfc747375d0ca in term 0.
I20260620 14:34:36.400744 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1f0def91464d5c95c6e59be05e2c1b" candidate_uuid: "80c82b4c075d4a32b76cfc747375d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493a182baebd4acf9199008ae7d8659e" is_pre_election: true
I20260620 14:34:36.400954 24366 leader_election.cc:302] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 590c4e463e6247afb6a861e92b92f869, 80c82b4c075d4a32b76cfc747375d0ca; no voters:
I20260620 14:34:36.401083 24168 raft_consensus.cc:2454] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c82b4c075d4a32b76cfc747375d0ca in term 0.
I20260620 14:34:36.401243 24501 raft_consensus.cc:2781] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:34:36.401340 24501 raft_consensus.cc:482] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:34:36.401391 24501 raft_consensus.cc:3037] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:36.402439 24501 raft_consensus.cc:504] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.402771 24501 leader_election.cc:288] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [CANDIDATE]: Term 1 election: Requested vote from peers 590c4e463e6247afb6a861e92b92f869 (127.4.58.66:37069), 493a182baebd4acf9199008ae7d8659e (127.4.58.65:42933)
I20260620 14:34:36.403240 24168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1f0def91464d5c95c6e59be05e2c1b" candidate_uuid: "80c82b4c075d4a32b76cfc747375d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493a182baebd4acf9199008ae7d8659e"
I20260620 14:34:36.403249 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d1f0def91464d5c95c6e59be05e2c1b" candidate_uuid: "80c82b4c075d4a32b76cfc747375d0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "590c4e463e6247afb6a861e92b92f869"
I20260620 14:34:36.403369 24168 raft_consensus.cc:3037] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:36.403383 24299 raft_consensus.cc:3037] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:36.404429 24168 raft_consensus.cc:2454] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c82b4c075d4a32b76cfc747375d0ca in term 1.
I20260620 14:34:36.404438 24299 raft_consensus.cc:2454] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c82b4c075d4a32b76cfc747375d0ca in term 1.
I20260620 14:34:36.404775 24366 leader_election.cc:302] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493a182baebd4acf9199008ae7d8659e, 80c82b4c075d4a32b76cfc747375d0ca; no voters:
I20260620 14:34:36.404958 24501 raft_consensus.cc:2781] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:36.405170 24501 raft_consensus.cc:686] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 1 LEADER]: Becoming Leader. State: Replica: 80c82b4c075d4a32b76cfc747375d0ca, State: Running, Role: LEADER
I20260620 14:34:36.405503 24501 consensus_queue.cc:237] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } }
I20260620 14:34:36.408713 24025 catalog_manager.cc:5495] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c82b4c075d4a32b76cfc747375d0ca (127.4.58.67). New cstate: current_term: 1 leader_uuid: "80c82b4c075d4a32b76cfc747375d0ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c82b4c075d4a32b76cfc747375d0ca" member_type: VOTER last_known_addr { host: "127.4.58.67" port: 42219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 } health_report { overall_health: UNKNOWN } } }
W20260620 14:34:36.450663 24346 tablet.cc:2369] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:34:36.475900 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.472540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.4.58.67:42219)
0620 14:34:36.472650 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:36.472653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:36.472666 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:34:36.472741 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:36.472743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:36.472835 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:36.472924 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:34:36.473222 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.473710 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.474550 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:36.474756 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:36.474838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:36.475078 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:34:36.475537 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:34:36.475628 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:36.475678 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:34:36.475736 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:34:36.482033 24168 raft_consensus.cc:1261] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e [term 1 FOLLOWER]: Refusing update from remote peer 80c82b4c075d4a32b76cfc747375d0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:36.482033 24299 raft_consensus.cc:1261] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 1 FOLLOWER]: Refusing update from remote peer 80c82b4c075d4a32b76cfc747375d0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:34:36.482817 24501 consensus_queue.cc:1035] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "590c4e463e6247afb6a861e92b92f869" member_type: VOTER last_known_addr { host: "127.4.58.66" port: 37069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:36.483042 24503 consensus_queue.cc:1035] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "493a182baebd4acf9199008ae7d8659e" member_type: VOTER last_known_addr { host: "127.4.58.65" port: 42933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:34:36.492964 24507 mvcc.cc:205] Tried to move back new op lower bound from 7298933049262182400 to 7298933048959406080. Current Snapshot: MvccSnapshot[applied={T|T < 7298933049262182400}]
I20260620 14:34:36.502703 24026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:36.502868 24026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:34:36.515372 24026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 456441b7c2b14710a2baa3ddaae94996: Sending DeleteTablet for 3 replicas of tablet 9d1f0def91464d5c95c6e59be05e2c1b
I20260620 14:34:36.516505 24279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:36 UTC) from {username='slave'} at 127.0.0.1:57306
I20260620 14:34:36.516580 24148 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:36 UTC) from {username='slave'} at 127.0.0.1:37732
I20260620 14:34:36.517251 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24086
I20260620 14:34:36.517422 24410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d1f0def91464d5c95c6e59be05e2c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:34:36 UTC) from {username='slave'} at 127.0.0.1:53804
I20260620 14:34:36.517645 24516 tablet_replica.cc:330] T 9d1f0def91464d5c95c6e59be05e2c1b P 493a182baebd4acf9199008ae7d8659e: stopping tablet replica
I20260620 14:34:36.517524 24517 tablet_replica.cc:330] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: stopping tablet replica
I20260620 14:34:36.518124 24517 raft_consensus.cc:2229] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:34:36.518582 24517 raft_consensus.cc:2258] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:36.519013 24518 tablet_replica.cc:330] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: stopping tablet replica
I20260620 14:34:36.519536 24518 raft_consensus.cc:2229] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:34:36.520031 24518 raft_consensus.cc:2258] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:34:36.521832 24518 ts_tablet_manager.cc:1905] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:36.522511 24517 ts_tablet_manager.cc:1905] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:34:36.524713 24518 ts_tablet_manager.cc:1918] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:34:36.524919 24518 log.cc:1196] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-2/wal/wals/9d1f0def91464d5c95c6e59be05e2c1b
I20260620 14:34:36.525300 24518 ts_tablet_manager.cc:1939] T 9d1f0def91464d5c95c6e59be05e2c1b P 80c82b4c075d4a32b76cfc747375d0ca: Deleting consensus metadata
I20260620 14:34:36.525411 24517 ts_tablet_manager.cc:1918] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 14:34:36.525388 24013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.58.65:42933 (error 108)
I20260620 14:34:36.525516 24517 log.cc:1196] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Deleting WAL directory at /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965824702348-4329-0/minicluster-data/ts-1/wal/wals/9d1f0def91464d5c95c6e59be05e2c1b
I20260620 14:34:36.525835 24517 ts_tablet_manager.cc:1939] T 9d1f0def91464d5c95c6e59be05e2c1b P 590c4e463e6247afb6a861e92b92f869: Deleting consensus metadata
I20260620 14:34:36.526404 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24217
I20260620 14:34:36.527275 24013 catalog_manager.cc:4841] TS 80c82b4c075d4a32b76cfc747375d0ca (127.4.58.67:42219): tablet 9d1f0def91464d5c95c6e59be05e2c1b (table test-table [id=b6dae68a5c374f5b9376c3d4f67014ba]) successfully deleted
I20260620 14:34:36.528344 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.528007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.65:42933 (local address 127.0.0.1:37744)
0620 14:34:36.528168 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.528253 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.65:42933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:34:36.528589 24013 catalog_manager.cc:4584] TS 493a182baebd4acf9199008ae7d8659e (127.4.58.65:42933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d1f0def91464d5c95c6e59be05e2c1b: Network error: Client connection negotiation failed: client connection to 127.4.58.65:42933: connect: Connection refused (error 111)
I20260620 14:34:36.531946 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24348
I20260620 14:34:36.532176 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:36.531893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.66:37069 (local address 127.0.0.1:57322)
0620 14:34:36.532035 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:34:36.532092 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.66:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:34:36.537424 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 23995
2026-06-20T14:34:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1239 ms)
[----------] 6 tests from EncryptionPolicyTest (10458 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:36 dist-test-slave-gqzk krb5kdc[24522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:36 dist-test-slave-gqzk krb5kdc[24522](info): set up 2 sockets
Jun 20 14:34:36 dist-test-slave-gqzk krb5kdc[24522](info): commencing operation
krb5kdc: starting...
W20260620 14:34:37.500782 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:37.617545 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:37.728085 24535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:37.728370 24535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:37.728413 24535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:37.732487 24535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:37.732581 24535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:37.732609 24535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:37.732626 24535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:37.732657 24535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:37.738158 24535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:37.739280 24535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:37.740335 24535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:37.748845 24543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:37.748875 24540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:37.748951 24535 server_base.cc:1034] running on GCE node
W20260620 14:34:37.748898 24541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:37.749653 24535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:37.750725 24535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:37.751900 24535 hybrid_clock.cc:648] HybridClock initialized: now 1781966077751881 us; error 31 us; skew 500 ppm
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:37.754542 24535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:37.755415 24535 webserver.cc:460] Webserver started at http://127.0.0.1:35913/ using document root <none> and password file <none>
I20260620 14:34:37.755951 24535 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:37.756016 24535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:37.756208 24535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:37.758315 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "08fca000e8e34de1b5f984c6ac6e5ac3"
format_stamp: "Formatted at 2026-06-20 14:34:37 on dist-test-slave-gqzk"
server_key: "b9dd395e3701bf6ff698fd0064b40dea"
server_key_iv: "793e387af2b67f8b7620b3cf18dd4f1c"
server_key_version: "encryptionkey@0"
I20260620 14:34:37.758876 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "08fca000e8e34de1b5f984c6ac6e5ac3"
format_stamp: "Formatted at 2026-06-20 14:34:37 on dist-test-slave-gqzk"
server_key: "b9dd395e3701bf6ff698fd0064b40dea"
server_key_iv: "793e387af2b67f8b7620b3cf18dd4f1c"
server_key_version: "encryptionkey@0"
I20260620 14:34:37.762684 24535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:34:37.765285 24550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:37.766528 24535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:37.766649 24535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "08fca000e8e34de1b5f984c6ac6e5ac3"
format_stamp: "Formatted at 2026-06-20 14:34:37 on dist-test-slave-gqzk"
server_key: "b9dd395e3701bf6ff698fd0064b40dea"
server_key_iv: "793e387af2b67f8b7620b3cf18dd4f1c"
server_key_version: "encryptionkey@0"
I20260620 14:34:37.766755 24535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:37.783142 24535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:37.787498 24535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:37.787690 24535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:37.794785 24535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36921
I20260620 14:34:37.794826 24602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36921 every 8 connection(s)
I20260620 14:34:37.795743 24535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:37.798671 24603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:37.803788 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24535
I20260620 14:34:37.803943 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:37.804270 4329 external_mini_cluster.cc:1428] Setting key 93f713741d2b9545dcb2d72a4e9e27c0
I20260620 14:34:37.804826 24603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3: Bootstrap starting.
I20260620 14:34:37.807305 24603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:37.808034 24603 log.cc:824] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:37.809988 24603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3: No bootstrap required, opened a new log
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:34:37.813812 24603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 14:34:37.814061 24603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:37.814129 24603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fca000e8e34de1b5f984c6ac6e5ac3, State: Initialized, Role: FOLLOWER
I20260620 14:34:37.814610 24603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 14:34:37.814775 24603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:37.814848 24603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:37.814947 24603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:37.815809 24603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 14:34:37.816136 24603 leader_election.cc:302] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08fca000e8e34de1b5f984c6ac6e5ac3; no voters:
I20260620 14:34:37.816427 24603 leader_election.cc:288] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:37.816515 24608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:37.816802 24608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [term 1 LEADER]: Becoming Leader. State: Replica: 08fca000e8e34de1b5f984c6ac6e5ac3, State: Running, Role: LEADER
I20260620 14:34:37.817111 24608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 14:34:37.817797 24603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:37.818216 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08fca000e8e34de1b5f984c6ac6e5ac3. Latest consensus state: current_term: 1 leader_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260620 14:34:37.818414 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:37.818856 24613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:37.819147 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:37.805791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42867 (local address 127.0.0.1:36921)
0620 14:34:37.806246 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:37.806261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:37.806300 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:34:37.806979 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:37.806997 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:37.807348 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:37.807685 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:37.808515 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:37.809552 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:37.810443 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:37.810923 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:37.811074 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:37.813749 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:37.813754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:37.813768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:37.813796 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:37.815585 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:37.816108 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:37.816114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:37.816119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:37.816253 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:37.816630 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:37.816633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:37.816635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:37.817168 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:37.817396 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:37.817891 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:34:37.818121 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":154,"threads_started":1}
I20260620 14:34:37.821092 24609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08fca000e8e34de1b5f984c6ac6e5ac3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260620 14:34:37.821192 24609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:37.822450 24613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:37.822718 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:37.827950 24613 catalog_manager.cc:1347] Generated new cluster ID: 45bab7fbdaa64b888469044a6c5761d4
I20260620 14:34:37.828068 24613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:37.832975 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:37.824552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39337 (local address 127.0.0.1:36921)
0620 14:34:37.824637 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:37.824641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:37.824974 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:34:37.825081 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:37.825084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:37.825126 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:37.825190 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:37.825889 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:37.826528 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:37.827337 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:37.827569 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:37.827658 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:37.829403 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:37.829407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:37.829410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:37.829437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:37.831014 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:37.831521 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:37.831525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:37.831527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:37.831625 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:37.831915 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:37.831918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:37.831920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:37.832135 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:37.832281 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:37.832506 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:34:37.832815 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:34:37.854844 24613 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:37.856153 24613 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:37.867841 24613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08fca000e8e34de1b5f984c6ac6e5ac3: Generated new TSK 0
I20260620 14:34:37.868729 24613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:37.873498 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24535
W20260620 14:34:37.878482 24520 connection.cc:535] client connection to 127.0.0.1:36921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36921): Connection reset by peer (error 104)
2026-06-20T14:34:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24634](info): set up 2 sockets
Jun 20 14:34:37 dist-test-slave-gqzk krb5kdc[24634](info): commencing operation
krb5kdc: starting...
W20260620 14:34:38.828668 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:38 dist-test-slave-gqzk krb5kdc[24634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:38.945423 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:39.053992 24647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:39.054272 24647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:39.054342 24647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:39.058372 24647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:39.058439 24647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:39.058465 24647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:39.058485 24647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:39.058507 24647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:39.063714 24647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:39.064767 24647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:39.065773 24647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:39.074033 24653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:39.074021 24655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:39.074021 24652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:39.074278 24647 server_base.cc:1034] running on GCE node
I20260620 14:34:39.074939 24647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:39.076053 24647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:39.077271 24647 hybrid_clock.cc:648] HybridClock initialized: now 1781966079077242 us; error 45 us; skew 500 ppm
Jun 20 14:34:39 dist-test-slave-gqzk krb5kdc[24634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:39.079922 24647 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:39.080888 24647 webserver.cc:460] Webserver started at http://127.0.0.1:42259/ using document root <none> and password file <none>
I20260620 14:34:39.081425 24647 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:39.081491 24647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:39.081679 24647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:39.083774 24647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7"
format_stamp: "Formatted at 2026-06-20 14:34:39 on dist-test-slave-gqzk"
server_key: "0eec0256725f6849b526b214f270489d"
server_key_iv: "b2888e0efea74b2a587f28202245ccf1"
server_key_version: "encryptionkey@0"
I20260620 14:34:39.084347 24647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7"
format_stamp: "Formatted at 2026-06-20 14:34:39 on dist-test-slave-gqzk"
server_key: "0eec0256725f6849b526b214f270489d"
server_key_iv: "b2888e0efea74b2a587f28202245ccf1"
server_key_version: "encryptionkey@0"
I20260620 14:34:39.088088 24647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 14:34:39.090728 24662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:39.092087 24647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:34:39.092271 24647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7"
format_stamp: "Formatted at 2026-06-20 14:34:39 on dist-test-slave-gqzk"
server_key: "0eec0256725f6849b526b214f270489d"
server_key_iv: "b2888e0efea74b2a587f28202245ccf1"
server_key_version: "encryptionkey@0"
I20260620 14:34:39.092387 24647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:39.122694 24647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:39.125823 24647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:39.126010 24647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:39.132997 24647 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43563
I20260620 14:34:39.133005 24714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43563 every 8 connection(s)
I20260620 14:34:39.133940 24647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:39.136713 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:39.141638 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24647
I20260620 14:34:39.141848 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:39.142218 4329 external_mini_cluster.cc:1428] Setting key 24c6287c587542639f0c983ed85a62b7
I20260620 14:34:39.142254 24715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: Bootstrap starting.
I20260620 14:34:39.144671 24715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:39.145423 24715 log.cc:824] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:39.147526 24715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: No bootstrap required, opened a new log
I20260620 14:34:39.151331 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:39.143578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50749 (local address 127.0.0.1:43563)
0620 14:34:39.144010 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:39.144028 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:39.144067 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:34:39.144734 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:39.144782 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:39.145149 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:39.145448 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:39.145844 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:39.146969 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:39.148134 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:39.148605 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:39.148729 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:39.149640 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:39.149645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:39.149658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:39.150053 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:39.150106 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:39.150129 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:39.150226 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:39.150420 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:34:39.150549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260620 14:34:39.151435 24715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } }
I20260620 14:34:39.151687 24715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:39.151741 24715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7b2f3c728a4410ac3004f7d9efb3f7, State: Initialized, Role: FOLLOWER
I20260620 14:34:39.152180 24715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } }
I20260620 14:34:39.152309 24715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:39.152374 24715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:39.152467 24715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:39.153343 24715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } }
I20260620 14:34:39.153647 24715 leader_election.cc:302] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f7b2f3c728a4410ac3004f7d9efb3f7; no voters:
I20260620 14:34:39.153923 24715 leader_election.cc:288] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:39.153995 24720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:39.154206 24720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f7b2f3c728a4410ac3004f7d9efb3f7, State: Running, Role: LEADER
I20260620 14:34:39.154240 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:39.154500 24720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } }
I20260620 14:34:39.155197 24715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:39.156073 24721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } } }
I20260620 14:34:39.156188 24721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:39.156451 24722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f7b2f3c728a4410ac3004f7d9efb3f7. Latest consensus state: current_term: 1 leader_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7b2f3c728a4410ac3004f7d9efb3f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43563 } } }
I20260620 14:34:39.156533 24722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7 [sys.catalog]: This master's current role is: LEADER
W20260620 14:34:39.159878 24740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:39.159950 24740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:39.160218 24734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:39.160869 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:39.156551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45721 (local address 127.0.0.1:43563)
0620 14:34:39.156816 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:39.156820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:39.156834 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:34:39.156868 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:39.156871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:39.156911 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:39.157032 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:39.157348 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:39.158073 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:39.159145 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:39.159465 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:39.159950 (+ 485us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:39.160432 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:39.160434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:39.160436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:39.160447 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:39.160468 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:39.160478 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:39.160546 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:39.160691 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:34:39.160724 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 14:34:39.161096 24734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:39.166705 24734 catalog_manager.cc:1347] Generated new cluster ID: 8a698b56e2b245fea2aa31a286c640c4
I20260620 14:34:39.166774 24734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:39.188629 24734 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:39.189632 24734 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:39.195255 24734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f7b2f3c728a4410ac3004f7d9efb3f7: Generated new TSK 0
I20260620 14:34:39.196126 24734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:39.209344 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24647
W20260620 14:34:39.214216 24632 connection.cc:535] client connection to 127.0.0.1:43563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43563): Connection reset by peer (error 104)
2026-06-20T14:34:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:39 dist-test-slave-gqzk krb5kdc[24746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:39 dist-test-slave-gqzk krb5kdc[24746](info): set up 2 sockets
Jun 20 14:34:39 dist-test-slave-gqzk krb5kdc[24746](info): commencing operation
krb5kdc: starting...
W20260620 14:34:40.157066 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:40 dist-test-slave-gqzk krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:40.274993 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:40.385504 24759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:40.385767 24759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:40.385807 24759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:40.389793 24759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:40.389860 24759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:40.389879 24759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:40.389896 24759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:40.389910 24759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:40.395180 24759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:40.396327 24759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:40.397339 24759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:40.405941 24767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:40.405946 24764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:40.406150 24759 server_base.cc:1034] running on GCE node
W20260620 14:34:40.405946 24765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:40.406919 24759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:40.408100 24759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:40.409274 24759 hybrid_clock.cc:648] HybridClock initialized: now 1781966080409240 us; error 43 us; skew 500 ppm
Jun 20 14:34:40 dist-test-slave-gqzk krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:40.412046 24759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:40.412977 24759 webserver.cc:460] Webserver started at http://127.0.0.1:46609/ using document root <none> and password file <none>
I20260620 14:34:40.413523 24759 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:40.413592 24759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:40.413784 24759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:40.415786 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "78cca70483404955a063bff5e634acb9"
format_stamp: "Formatted at 2026-06-20 14:34:40 on dist-test-slave-gqzk"
server_key: "a66992c98d2cf3dd0b929d987cb1d883"
server_key_iv: "0fdc8c931f438a11b26a36bbbb7763a6"
server_key_version: "encryptionkey@0"
I20260620 14:34:40.416306 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "78cca70483404955a063bff5e634acb9"
format_stamp: "Formatted at 2026-06-20 14:34:40 on dist-test-slave-gqzk"
server_key: "a66992c98d2cf3dd0b929d987cb1d883"
server_key_iv: "0fdc8c931f438a11b26a36bbbb7763a6"
server_key_version: "encryptionkey@0"
I20260620 14:34:40.419977 24759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:34:40.422617 24774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:40.423903 24759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:34:40.424045 24759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "78cca70483404955a063bff5e634acb9"
format_stamp: "Formatted at 2026-06-20 14:34:40 on dist-test-slave-gqzk"
server_key: "a66992c98d2cf3dd0b929d987cb1d883"
server_key_iv: "0fdc8c931f438a11b26a36bbbb7763a6"
server_key_version: "encryptionkey@0"
I20260620 14:34:40.424165 24759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:40.428642 24759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:40.431718 24759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:40.431891 24759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:40.438479 24759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46355
I20260620 14:34:40.438516 24826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46355 every 8 connection(s)
I20260620 14:34:40.439415 24759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:40.440836 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24759
I20260620 14:34:40.440965 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:40.441252 4329 external_mini_cluster.cc:1428] Setting key 8c43b8e3a706d9f721b8b7b2569bf2a9
I20260620 14:34:40.442421 24827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:40.446672 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:40.442596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40527 (local address 127.0.0.1:46355)
0620 14:34:40.442998 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:40.443017 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:40.443059 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:34:40.443882 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:40.443906 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:40.444249 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:40.444619 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:40.445283 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:40.445289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:40.445313 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:40.445324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:40.445382 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:40.445407 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:40.445555 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:40.445622 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:34:40.445769 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260620 14:34:40.448837 24827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9: Bootstrap starting.
I20260620 14:34:40.449486 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:40.451092 24827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:40.451902 24827 log.cc:824] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:40.453023 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:40.451443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55865 (local address 127.0.0.1:46355)
0620 14:34:40.451546 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:40.451551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:40.451741 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:34:40.451868 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:40.451871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:40.451906 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:40.451969 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:40.452634 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:40.452637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:40.452640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:40.452652 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:40.452679 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:40.452689 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:40.452793 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:40.452873 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:34:40.452911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:34:40.453791 24827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9: No bootstrap required, opened a new log
I20260620 14:34:40.457317 24827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260620 14:34:40.457544 24827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:40.457584 24827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78cca70483404955a063bff5e634acb9, State: Initialized, Role: FOLLOWER
I20260620 14:34:40.457976 24827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260620 14:34:40.458112 24827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:40.458164 24827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:40.458236 24827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:40.459077 24827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260620 14:34:40.459434 24827 leader_election.cc:302] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78cca70483404955a063bff5e634acb9; no voters:
I20260620 14:34:40.459757 24827 leader_election.cc:288] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:40.459822 24837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:40.460033 24837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [term 1 LEADER]: Becoming Leader. State: Replica: 78cca70483404955a063bff5e634acb9, State: Running, Role: LEADER
I20260620 14:34:40.460336 24837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260620 14:34:40.461117 24827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:40.462095 24839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78cca70483404955a063bff5e634acb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } } }
I20260620 14:34:40.462124 24838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78cca70483404955a063bff5e634acb9. Latest consensus state: current_term: 1 leader_uuid: "78cca70483404955a063bff5e634acb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78cca70483404955a063bff5e634acb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } } }
I20260620 14:34:40.462216 24839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:40.462220 24838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:40.462646 24846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:40.465415 24846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:40.470861 24846 catalog_manager.cc:1347] Generated new cluster ID: 8c617a28c3474982b64232a3ab3d8260
I20260620 14:34:40.470942 24846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:40.501605 24846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:40.501960 24846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:40.518433 24846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78cca70483404955a063bff5e634acb9: Generated new TSK 0
I20260620 14:34:40.519217 24846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:40.534009 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24759
2026-06-20T14:34:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:40 dist-test-slave-gqzk krb5kdc[24858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:40 dist-test-slave-gqzk krb5kdc[24858](info): set up 2 sockets
Jun 20 14:34:40 dist-test-slave-gqzk krb5kdc[24858](info): commencing operation
krb5kdc: starting...
W20260620 14:34:41.496690 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:41.613550 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:41.722265 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:41.722546 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:41.722591 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:41.726565 24871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:41.726627 24871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:41.726645 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:41.726660 24871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:41.726675 24871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:41.732048 24871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:41.733107 24871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:41.734099 24871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:41.742837 24876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:41.742894 24879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:41.743003 24871 server_base.cc:1034] running on GCE node
W20260620 14:34:41.742837 24877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:41.743827 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:41.745064 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:41.746229 24871 hybrid_clock.cc:648] HybridClock initialized: now 1781966081746210 us; error 37 us; skew 500 ppm
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:41.752156 24871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:41.753223 24871 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260620 14:34:41.753793 24871 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:41.753856 24871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:41.754046 24871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:41.756138 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88"
format_stamp: "Formatted at 2026-06-20 14:34:41 on dist-test-slave-gqzk"
server_key: "0f1ee330fefb25b78d4f94401d13d617"
server_key_iv: "556baab67972c6a97a4b04105744bca2"
server_key_version: "encryptionkey@0"
I20260620 14:34:41.756695 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88"
format_stamp: "Formatted at 2026-06-20 14:34:41 on dist-test-slave-gqzk"
server_key: "0f1ee330fefb25b78d4f94401d13d617"
server_key_iv: "556baab67972c6a97a4b04105744bca2"
server_key_version: "encryptionkey@0"
I20260620 14:34:41.760553 24871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:34:41.763192 24886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:41.764581 24871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:34:41.764734 24871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88"
format_stamp: "Formatted at 2026-06-20 14:34:41 on dist-test-slave-gqzk"
server_key: "0f1ee330fefb25b78d4f94401d13d617"
server_key_iv: "556baab67972c6a97a4b04105744bca2"
server_key_version: "encryptionkey@0"
I20260620 14:34:41.764894 24871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:41.780174 24871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:41.783655 24871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:41.783854 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:41.790858 24871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43325
I20260620 14:34:41.790887 24938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43325 every 8 connection(s)
I20260620 14:34:41.791846 24871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:41.794725 24939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:41.799585 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24871
I20260620 14:34:41.799716 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:41.799999 4329 external_mini_cluster.cc:1428] Setting key 2534c91ad4d10f9da765be6a3739fc3d
I20260620 14:34:41.800443 24939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88: Bootstrap starting.
I20260620 14:34:41.802968 24939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:41.803930 24939 log.cc:824] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:41.806089 24939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88: No bootstrap required, opened a new log
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:34:41.810355 24939 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } }
I20260620 14:34:41.810593 24939 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:41.810652 24939 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cd10e2d0e14d03a2b31c2bd239fb88, State: Initialized, Role: FOLLOWER
I20260620 14:34:41.811091 24939 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } }
I20260620 14:34:41.811218 24939 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:41.811280 24939 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:41.811370 24939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:41.812275 24939 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } }
I20260620 14:34:41.812605 24939 leader_election.cc:302] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6cd10e2d0e14d03a2b31c2bd239fb88; no voters:
I20260620 14:34:41.812947 24939 leader_election.cc:288] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:41.813052 24944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:41.813251 24944 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [term 1 LEADER]: Becoming Leader. State: Replica: e6cd10e2d0e14d03a2b31c2bd239fb88, State: Running, Role: LEADER
I20260620 14:34:41.813634 24944 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } }
I20260620 14:34:41.814527 24939 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:41.814728 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:41.801374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58649 (local address 127.0.0.1:43325)
0620 14:34:41.801827 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:41.801842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:41.801879 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:34:41.802523 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:41.802542 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:41.802908 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:41.803215 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:41.804211 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:41.805283 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:41.806475 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:41.806949 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:41.807068 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:41.809738 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:41.809745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:41.809765 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:41.809801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:41.811527 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:41.811954 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:41.811960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:41.811966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:41.812084 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:41.812343 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:41.812346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:41.812348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:41.812830 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:41.813031 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:41.813628 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 14:34:41.813946 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260620 14:34:41.815532 24945 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6cd10e2d0e14d03a2b31c2bd239fb88. Latest consensus state: current_term: 1 leader_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } } }
I20260620 14:34:41.815670 24945 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:41.815977 24946 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6cd10e2d0e14d03a2b31c2bd239fb88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43325 } } }
I20260620 14:34:41.816071 24946 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:41.816960 24953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:41.818140 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:41.820073 24953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:41.826143 24953 catalog_manager.cc:1347] Generated new cluster ID: c1a7809bf46f4cc6b7eca092115060c1
I20260620 14:34:41.826215 24953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:41.837970 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:41.819940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42799 (local address 127.0.0.1:43325)
0620 14:34:41.820044 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:41.820048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:41.822466 (+ 2418us) server_negotiation.cc:407] Connection header received
0620 14:34:41.822596 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:41.822599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:41.822650 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:41.822739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:41.823568 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:41.824209 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:41.825191 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:41.825416 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:41.825500 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:41.834332 (+ 8832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:41.834337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:41.834344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:41.834391 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:41.836090 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:41.836645 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:41.836649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:41.836650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:41.836778 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:41.837061 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:41.837065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:41.837067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:41.837305 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:41.837429 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:41.837656 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:34:41.837784 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:34:41.841130 24953 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:41.842272 24953 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:41.863142 24953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6cd10e2d0e14d03a2b31c2bd239fb88: Generated new TSK 0
I20260620 14:34:41.863934 24953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:41.868161 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24871
2026-06-20T14:34:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24970](info): set up 2 sockets
Jun 20 14:34:41 dist-test-slave-gqzk krb5kdc[24970](info): commencing operation
krb5kdc: starting...
W20260620 14:34:42.824651 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:42 dist-test-slave-gqzk krb5kdc[24970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:42.944214 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:43.054023 24983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:43.054347 24983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:43.054410 24983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:43.058522 24983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:43.058597 24983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:43.058622 24983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:43.058642 24983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:43.058663 24983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:43.064100 24983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:43.065198 24983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:43.066217 24983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:43.074991 24991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:43.075004 24988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:43.075064 24983 server_base.cc:1034] running on GCE node
W20260620 14:34:43.075004 24989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:43.075866 24983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:43.077134 24983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:43.078299 24983 hybrid_clock.cc:648] HybridClock initialized: now 1781966083078273 us; error 43 us; skew 500 ppm
Jun 20 14:34:43 dist-test-slave-gqzk krb5kdc[24970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:43.081058 24983 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:43.081971 24983 webserver.cc:460] Webserver started at http://127.0.0.1:37539/ using document root <none> and password file <none>
I20260620 14:34:43.082511 24983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:43.082568 24983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:43.082736 24983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:43.084896 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "d957927a73644ecbb2c6d30926b9608e"
format_stamp: "Formatted at 2026-06-20 14:34:43 on dist-test-slave-gqzk"
server_key: "fb737df8be14fc43ede715b41f187622"
server_key_iv: "f4f7000c643727da0556e37682119d6a"
server_key_version: "encryptionkey@0"
I20260620 14:34:43.085454 24983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "d957927a73644ecbb2c6d30926b9608e"
format_stamp: "Formatted at 2026-06-20 14:34:43 on dist-test-slave-gqzk"
server_key: "fb737df8be14fc43ede715b41f187622"
server_key_iv: "f4f7000c643727da0556e37682119d6a"
server_key_version: "encryptionkey@0"
I20260620 14:34:43.089206 24983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:34:43.091992 24998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:43.093441 24983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:43.093554 24983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "d957927a73644ecbb2c6d30926b9608e"
format_stamp: "Formatted at 2026-06-20 14:34:43 on dist-test-slave-gqzk"
server_key: "fb737df8be14fc43ede715b41f187622"
server_key_iv: "f4f7000c643727da0556e37682119d6a"
server_key_version: "encryptionkey@0"
I20260620 14:34:43.093657 24983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:43.114102 24983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:43.117214 24983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:43.117376 24983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:43.124351 24983 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43781
I20260620 14:34:43.124431 25050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43781 every 8 connection(s)
I20260620 14:34:43.125355 24983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:43.128171 25051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:43.130718 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 24983
I20260620 14:34:43.130836 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:43.131083 4329 external_mini_cluster.cc:1428] Setting key d15957d2943ed669c7cd3f9e35325c08
I20260620 14:34:43.134019 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: Bootstrap starting.
I20260620 14:34:43.136168 25051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:43.136931 25051 log.cc:824] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:43.138827 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: No bootstrap required, opened a new log
I20260620 14:34:43.139816 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:43.132491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41169 (local address 127.0.0.1:43781)
0620 14:34:43.132955 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:43.132969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:43.133006 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:34:43.133635 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:43.133653 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:43.134011 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:43.134335 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:43.134750 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:43.135724 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:43.136655 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:43.137201 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:43.137348 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:43.138159 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:43.138164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:43.138177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:43.138564 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:43.138621 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:43.138644 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:43.138730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:43.138871 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:34:43.139091 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":118,"threads_started":1}
I20260620 14:34:43.142534 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:43.142362 25051 raft_consensus.cc:348] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260620 14:34:43.142613 25051 raft_consensus.cc:374] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:43.142668 25051 raft_consensus.cc:729] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d957927a73644ecbb2c6d30926b9608e, State: Initialized, Role: FOLLOWER
I20260620 14:34:43.143136 25051 consensus_queue.cc:260] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260620 14:34:43.143296 25051 raft_consensus.cc:388] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:43.143373 25051 raft_consensus.cc:482] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:43.143473 25051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:43.144626 25051 raft_consensus.cc:504] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260620 14:34:43.145015 25051 leader_election.cc:302] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d957927a73644ecbb2c6d30926b9608e; no voters:
I20260620 14:34:43.145347 25051 leader_election.cc:288] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:43.145445 25061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:43.145730 25061 raft_consensus.cc:686] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [term 1 LEADER]: Becoming Leader. State: Replica: d957927a73644ecbb2c6d30926b9608e, State: Running, Role: LEADER
I20260620 14:34:43.146118 25061 consensus_queue.cc:237] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260620 14:34:43.146687 25051 sys_catalog.cc:564] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:43.147361 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d957927a73644ecbb2c6d30926b9608e. Latest consensus state: current_term: 1 leader_uuid: "d957927a73644ecbb2c6d30926b9608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260620 14:34:43.147482 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:43.148422 25062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d957927a73644ecbb2c6d30926b9608e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957927a73644ecbb2c6d30926b9608e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260620 14:34:43.148581 25062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:43.149797 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:43.144947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60945 (local address 127.0.0.1:43781)
0620 14:34:43.145072 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:43.145076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:43.145404 (+ 328us) server_negotiation.cc:407] Connection header received
0620 14:34:43.145532 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:43.145535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:43.145589 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:43.145673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:43.146058 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:43.146860 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:43.147989 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:43.148300 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:43.148395 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:43.149363 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:43.149366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:43.149370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:43.149384 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:43.149414 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:43.149428 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:43.149470 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:43.149608 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:34:43.149662 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:34:43.151809 25076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:43.151886 25076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:43.152132 25067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:43.153053 25067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:43.158780 25067 catalog_manager.cc:1347] Generated new cluster ID: ccb1b1dc030d41d19dbe4eeab46edfa1
I20260620 14:34:43.158841 25067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:43.165083 25067 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:43.166181 25067 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:43.173416 25067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d957927a73644ecbb2c6d30926b9608e: Generated new TSK 0
I20260620 14:34:43.174274 25067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:43.184051 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 24983
2026-06-20T14:34:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:43 dist-test-slave-gqzk krb5kdc[25082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:43 dist-test-slave-gqzk krb5kdc[25082](info): set up 2 sockets
Jun 20 14:34:43 dist-test-slave-gqzk krb5kdc[25082](info): commencing operation
krb5kdc: starting...
W20260620 14:34:44.136744 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:44 dist-test-slave-gqzk krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:44.254057 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:44.366518 25095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:44.366817 25095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:44.366882 25095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:44.371006 25095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:44.371083 25095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:44.371117 25095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:44.371155 25095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:44.371186 25095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:44.376543 25095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:44.377641 25095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:44.378690 25095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:44.387065 25101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:44.387066 25100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:44.387066 25103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:44.387486 25095 server_base.cc:1034] running on GCE node
I20260620 14:34:44.388031 25095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:44.389158 25095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:44.390317 25095 hybrid_clock.cc:648] HybridClock initialized: now 1781966084390295 us; error 35 us; skew 500 ppm
Jun 20 14:34:44 dist-test-slave-gqzk krb5kdc[25082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:44.393013 25095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:44.393885 25095 webserver.cc:460] Webserver started at http://127.0.0.1:39127/ using document root <none> and password file <none>
I20260620 14:34:44.394407 25095 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:44.394472 25095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:44.394654 25095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:44.396660 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "4c3e808492cb45a0ad0c2b98067ef945"
format_stamp: "Formatted at 2026-06-20 14:34:44 on dist-test-slave-gqzk"
server_key: "47fb43571bc167f9ee35376617175c44"
server_key_iv: "87d1ee1e7b1017557753d94079edb578"
server_key_version: "encryptionkey@0"
I20260620 14:34:44.397248 25095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "4c3e808492cb45a0ad0c2b98067ef945"
format_stamp: "Formatted at 2026-06-20 14:34:44 on dist-test-slave-gqzk"
server_key: "47fb43571bc167f9ee35376617175c44"
server_key_iv: "87d1ee1e7b1017557753d94079edb578"
server_key_version: "encryptionkey@0"
I20260620 14:34:44.401108 25095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:34:44.403673 25110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:44.405035 25095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:44.405161 25095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "4c3e808492cb45a0ad0c2b98067ef945"
format_stamp: "Formatted at 2026-06-20 14:34:44 on dist-test-slave-gqzk"
server_key: "47fb43571bc167f9ee35376617175c44"
server_key_iv: "87d1ee1e7b1017557753d94079edb578"
server_key_version: "encryptionkey@0"
I20260620 14:34:44.405273 25095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:44.409946 25095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:44.413159 25095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:44.413338 25095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:44.420353 25095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34081
I20260620 14:34:44.420367 25162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34081 every 8 connection(s)
I20260620 14:34:44.421350 25095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:44.424091 25163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:44.429515 25163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945: Bootstrap starting.
I20260620 14:34:44.430308 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25095
I20260620 14:34:44.430442 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:44.430733 4329 external_mini_cluster.cc:1428] Setting key 6dd1697d31eb4dd3c41f1d4c3d3d766e
I20260620 14:34:44.432343 25163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:44.433068 25163 log.cc:824] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:44.434932 25163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945: No bootstrap required, opened a new log
I20260620 14:34:44.435705 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:44.431929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52009 (local address 127.0.0.1:34081)
0620 14:34:44.432371 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:44.432385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:44.432430 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:34:44.433122 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:44.433144 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:44.433451 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:44.433763 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:44.434531 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:44.434537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:44.434549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:44.434559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:44.434610 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:44.434629 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:44.434725 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:44.434851 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 14:34:44.435006 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260620 14:34:44.438406 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:44.438473 25163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } }
I20260620 14:34:44.438706 25163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:44.438759 25163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3e808492cb45a0ad0c2b98067ef945, State: Initialized, Role: FOLLOWER
I20260620 14:34:44.439136 25163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } }
I20260620 14:34:44.439246 25163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:44.439299 25163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:44.439384 25163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:44.440294 25163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } }
I20260620 14:34:44.440653 25163 leader_election.cc:302] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c3e808492cb45a0ad0c2b98067ef945; no voters:
I20260620 14:34:44.440971 25163 leader_election.cc:288] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:44.441071 25173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:44.441272 25173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [term 1 LEADER]: Becoming Leader. State: Replica: 4c3e808492cb45a0ad0c2b98067ef945, State: Running, Role: LEADER
I20260620 14:34:44.441561 25173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } }
I20260620 14:34:44.442281 25163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:44.443130 25175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c3e808492cb45a0ad0c2b98067ef945. Latest consensus state: current_term: 1 leader_uuid: "4c3e808492cb45a0ad0c2b98067ef945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } } }
I20260620 14:34:44.443392 25174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c3e808492cb45a0ad0c2b98067ef945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e808492cb45a0ad0c2b98067ef945" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34081 } } }
I20260620 14:34:44.443496 25175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:44.443593 25174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:44.444478 25184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:44.444907 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:44.440458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53901 (local address 127.0.0.1:34081)
0620 14:34:44.440552 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:44.440556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:44.440768 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:34:44.440885 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:44.440888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:44.440920 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:44.440980 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:44.444461 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:44.444465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:44.444468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:44.444480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:44.444508 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:44.444520 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:44.444609 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:44.444733 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:34:44.444792 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:34:44.447613 25184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:44.452569 25184 catalog_manager.cc:1347] Generated new cluster ID: b399d530751045369bdc83c79564ad8a
I20260620 14:34:44.452634 25184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:44.469188 25184 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:44.469535 25184 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:44.479462 25184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c3e808492cb45a0ad0c2b98067ef945: Generated new TSK 0
I20260620 14:34:44.480340 25184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:44.486460 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25095
2026-06-20T14:34:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:44 dist-test-slave-gqzk krb5kdc[25194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:44 dist-test-slave-gqzk krb5kdc[25194](info): set up 2 sockets
Jun 20 14:34:44 dist-test-slave-gqzk krb5kdc[25194](info): commencing operation
krb5kdc: starting...
W20260620 14:34:45.444798 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:45.563800 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:45.673188 25207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:45.673460 25207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:45.673501 25207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:45.677462 25207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:45.677518 25207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:45.677536 25207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:45.677554 25207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:45.677582 25207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:45.682891 25207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:45.683967 25207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:45.685014 25207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:45.693423 25212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:45.693468 25207 server_base.cc:1034] running on GCE node
W20260620 14:34:45.693432 25215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:45.693468 25213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:45.694357 25207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:45.695485 25207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:45.696640 25207 hybrid_clock.cc:648] HybridClock initialized: now 1781966085696614 us; error 41 us; skew 500 ppm
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:45.699424 25207 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:45.700312 25207 webserver.cc:460] Webserver started at http://127.0.0.1:36025/ using document root <none> and password file <none>
I20260620 14:34:45.700873 25207 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:45.700942 25207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:45.701133 25207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:45.703140 25207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "9e1b5883650940b78d62102fbf412128"
format_stamp: "Formatted at 2026-06-20 14:34:45 on dist-test-slave-gqzk"
server_key: "6b042bf5882132442e2e4e92fdc0afc1"
server_key_iv: "2622b714470e13878a73fcc5c9eed44c"
server_key_version: "encryptionkey@0"
I20260620 14:34:45.703683 25207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "9e1b5883650940b78d62102fbf412128"
format_stamp: "Formatted at 2026-06-20 14:34:45 on dist-test-slave-gqzk"
server_key: "6b042bf5882132442e2e4e92fdc0afc1"
server_key_iv: "2622b714470e13878a73fcc5c9eed44c"
server_key_version: "encryptionkey@0"
I20260620 14:34:45.707376 25207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:34:45.709820 25222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:45.711119 25207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:45.711249 25207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "9e1b5883650940b78d62102fbf412128"
format_stamp: "Formatted at 2026-06-20 14:34:45 on dist-test-slave-gqzk"
server_key: "6b042bf5882132442e2e4e92fdc0afc1"
server_key_iv: "2622b714470e13878a73fcc5c9eed44c"
server_key_version: "encryptionkey@0"
I20260620 14:34:45.711351 25207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:45.736471 25207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:45.739362 25207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:45.739547 25207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:45.746315 25207 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42479
I20260620 14:34:45.746325 25274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42479 every 8 connection(s)
I20260620 14:34:45.747288 25207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:45.750165 25275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:45.750211 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25207
I20260620 14:34:45.750329 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:45.750638 4329 external_mini_cluster.cc:1428] Setting key 412e01dfa20b186e040464b8d7ea85eb
I20260620 14:34:45.757383 25275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: Bootstrap starting.
I20260620 14:34:45.760097 25275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: Neither blocks nor log segments found. Creating new log.
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:34:45.760857 25275 log.cc:824] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:45.762759 25275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: No bootstrap required, opened a new log
I20260620 14:34:45.765278 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:45.752109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:48503 (local address 127.0.0.1:42479)
0620 14:34:45.752810 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:45.752830 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:45.752868 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:34:45.753657 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:45.753675 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:45.754037 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:45.754327 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:45.755214 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:45.756239 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:45.757445 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:45.757969 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:45.758086 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:45.760738 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:45.760743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:45.760785 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:45.760819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:45.762450 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:45.762947 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:45.762952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:45.762957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:45.763064 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:45.763376 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:45.763378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:45.763379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:45.763765 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:45.763978 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:45.764272 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:34:45.764487 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":165,"threads_started":1}
I20260620 14:34:45.766391 25275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:34:45.766619 25275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:45.766674 25275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1b5883650940b78d62102fbf412128, State: Initialized, Role: FOLLOWER
I20260620 14:34:45.767125 25275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:34:45.767324 25275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:45.767395 25275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:45.767496 25275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:45.768208 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:45.768518 25275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:34:45.768904 25275 leader_election.cc:302] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e1b5883650940b78d62102fbf412128; no voters:
I20260620 14:34:45.769224 25275 leader_election.cc:288] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:45.769361 25280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:45.769626 25280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [term 1 LEADER]: Becoming Leader. State: Replica: 9e1b5883650940b78d62102fbf412128, State: Running, Role: LEADER
I20260620 14:34:45.770018 25280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } }
I20260620 14:34:45.770519 25275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:45.771191 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e1b5883650940b78d62102fbf412128. Latest consensus state: current_term: 1 leader_uuid: "9e1b5883650940b78d62102fbf412128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 14:34:45.771361 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:45.771911 25286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e1b5883650940b78d62102fbf412128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1b5883650940b78d62102fbf412128" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42479 } } }
I20260620 14:34:45.772037 25286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128 [sys.catalog]: This master's current role is: LEADER
W20260620 14:34:45.775753 25300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:45.775846 25300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:45.776010 25294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:45.776942 25294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:45.779845 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:45.770329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:58549 (local address 127.0.0.1:42479)
0620 14:34:45.770444 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:45.770448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:45.771834 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 14:34:45.772153 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:45.772157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:45.772211 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:45.772265 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:45.773420 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:45.774016 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:45.774813 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:45.775040 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:45.775128 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:45.776553 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:45.776557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:45.776560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:45.776583 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:45.777965 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:45.778448 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:45.778451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:45.778453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:45.778528 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:45.778833 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:45.778837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:45.778838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:45.779052 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:45.779184 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:45.779595 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:34:45.779696 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:34:45.783257 25294 catalog_manager.cc:1347] Generated new cluster ID: 8114c50ed8b34912b924d51e98791bd6
I20260620 14:34:45.783332 25294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:45.796520 25294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:45.797722 25294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:45.803843 25294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e1b5883650940b78d62102fbf412128: Generated new TSK 0
I20260620 14:34:45.804579 25294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:45.809263 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25207
W20260620 14:34:45.814093 25192 connection.cc:535] client connection to 127.0.0.1:42479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42479): Connection reset by peer (error 104)
2026-06-20T14:34:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25306](info): set up 2 sockets
Jun 20 14:34:45 dist-test-slave-gqzk krb5kdc[25306](info): commencing operation
krb5kdc: starting...
W20260620 14:34:46.760818 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:46 dist-test-slave-gqzk krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:46.881464 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:46.989874 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:46.990151 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:46.990195 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:46.994282 25319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:46.994349 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:46.994367 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:46.994383 25319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:46.994400 25319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:46.999784 25319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:47.000897 25319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:47.001914 25319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:34:47.010718 25319 server_base.cc:1034] running on GCE node
W20260620 14:34:47.010692 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:47.010667 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:47.010667 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:47.011423 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:47.012631 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:47.013809 25319 hybrid_clock.cc:648] HybridClock initialized: now 1781966087013775 us; error 34 us; skew 500 ppm
Jun 20 14:34:47 dist-test-slave-gqzk krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:47.022019 25319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:47.023051 25319 webserver.cc:460] Webserver started at http://127.0.0.1:44361/ using document root <none> and password file <none>
I20260620 14:34:47.023662 25319 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:47.023734 25319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:47.023942 25319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:47.026100 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "f7645fbbf07347178be7f7656e99cf7b"
format_stamp: "Formatted at 2026-06-20 14:34:47 on dist-test-slave-gqzk"
server_key: "25d43a6e9e7aa12f025b82fdfab19ddc"
server_key_iv: "8d06ccf9af980a0626efa37892f67616"
server_key_version: "encryptionkey@0"
I20260620 14:34:47.026664 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "f7645fbbf07347178be7f7656e99cf7b"
format_stamp: "Formatted at 2026-06-20 14:34:47 on dist-test-slave-gqzk"
server_key: "25d43a6e9e7aa12f025b82fdfab19ddc"
server_key_iv: "8d06ccf9af980a0626efa37892f67616"
server_key_version: "encryptionkey@0"
I20260620 14:34:47.030542 25319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:34:47.033227 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:47.034514 25319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:47.034652 25319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "f7645fbbf07347178be7f7656e99cf7b"
format_stamp: "Formatted at 2026-06-20 14:34:47 on dist-test-slave-gqzk"
server_key: "25d43a6e9e7aa12f025b82fdfab19ddc"
server_key_iv: "8d06ccf9af980a0626efa37892f67616"
server_key_version: "encryptionkey@0"
I20260620 14:34:47.034765 25319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:47.055393 25319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:47.058660 25319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:47.058858 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:47.065580 25319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37653
I20260620 14:34:47.065691 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37653 every 8 connection(s)
I20260620 14:34:47.066545 25319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:47.067745 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25319
I20260620 14:34:47.067871 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:47.068113 4329 external_mini_cluster.cc:1428] Setting key 0ffe1044b4508b052871a8d7d09bb7f6
I20260620 14:34:47.069585 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:47.076463 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b: Bootstrap starting.
I20260620 14:34:47.078238 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:47.069416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:53333 (local address 127.0.0.1:37653)
0620 14:34:47.069826 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:47.069844 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:47.069886 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:34:47.070690 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:47.070714 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:47.071157 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:47.071523 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:47.071944 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:47.073458 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:47.074386 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:47.075089 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:47.075261 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:47.076148 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:47.076154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:47.076171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:47.076651 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:47.076722 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:47.076798 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:47.076962 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:47.077143 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:34:47.077277 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260620 14:34:47.079136 25387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:47.079970 25387 log.cc:824] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:47.081275 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:47.081949 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b: No bootstrap required, opened a new log
I20260620 14:34:47.085476 25387 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260620 14:34:47.085758 25387 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:47.085834 25387 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7645fbbf07347178be7f7656e99cf7b, State: Initialized, Role: FOLLOWER
I20260620 14:34:47.086261 25387 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260620 14:34:47.086413 25387 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:47.086513 25387 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:47.086624 25387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:47.087608 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:47.083172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:50345 (local address 127.0.0.1:37653)
0620 14:34:47.083362 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:47.083365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:47.083415 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:34:47.083529 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:47.083531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:47.083568 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:47.083629 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:47.083950 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:47.084906 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:47.085957 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:47.086260 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:47.086346 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:47.087156 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:47.087160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:47.087162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:47.087174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:47.087239 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:47.087250 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:47.087318 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:47.087451 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:34:47.087481 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 14:34:47.087755 25387 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260620 14:34:47.088131 25387 leader_election.cc:302] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7645fbbf07347178be7f7656e99cf7b; no voters:
I20260620 14:34:47.088481 25387 leader_election.cc:288] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:47.088577 25397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:47.088848 25397 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [term 1 LEADER]: Becoming Leader. State: Replica: f7645fbbf07347178be7f7656e99cf7b, State: Running, Role: LEADER
I20260620 14:34:47.089172 25397 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260620 14:34:47.089756 25387 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:47.090739 25398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7645fbbf07347178be7f7656e99cf7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260620 14:34:47.090869 25398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:47.091184 25399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7645fbbf07347178be7f7656e99cf7b. Latest consensus state: current_term: 1 leader_uuid: "f7645fbbf07347178be7f7656e99cf7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7645fbbf07347178be7f7656e99cf7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260620 14:34:47.091315 25399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:47.091246 25406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:47.094430 25406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:47.100605 25406 catalog_manager.cc:1347] Generated new cluster ID: 2c483ccff80f4a08a48b582cc42a9bdb
I20260620 14:34:47.100687 25406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:47.115243 25406 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:47.116447 25406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:47.128206 25406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7645fbbf07347178be7f7656e99cf7b: Generated new TSK 0
I20260620 14:34:47.129220 25406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:47.138114 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25319
W20260620 14:34:47.143002 25304 connection.cc:535] client connection to 127.0.0.1:37653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37653): Connection reset by peer (error 104)
2026-06-20T14:34:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:47 dist-test-slave-gqzk krb5kdc[25418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:47 dist-test-slave-gqzk krb5kdc[25418](info): set up 2 sockets
Jun 20 14:34:47 dist-test-slave-gqzk krb5kdc[25418](info): commencing operation
krb5kdc: starting...
W20260620 14:34:48.092921 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:48 dist-test-slave-gqzk krb5kdc[25418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:48.211359 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:48.318132 25431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:48.318423 25431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:48.318481 25431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:48.322553 25431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:48.322629 25431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:48.322656 25431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:48.322676 25431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:48.322705 25431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:48.328197 25431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:48.329329 25431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:48.330345 25431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:48.338779 25437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:48.338779 25439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:48.338866 25436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:48.339016 25431 server_base.cc:1034] running on GCE node
I20260620 14:34:48.339682 25431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:48.340847 25431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:48.342011 25431 hybrid_clock.cc:648] HybridClock initialized: now 1781966088341988 us; error 43 us; skew 500 ppm
Jun 20 14:34:48 dist-test-slave-gqzk krb5kdc[25418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:48.344791 25431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:48.345677 25431 webserver.cc:460] Webserver started at http://127.0.0.1:38681/ using document root <none> and password file <none>
I20260620 14:34:48.346244 25431 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:48.346312 25431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:48.346511 25431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:48.348575 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "e5d207308ce0466ea94d5123137d1eea"
format_stamp: "Formatted at 2026-06-20 14:34:48 on dist-test-slave-gqzk"
server_key: "36dd68f8d02d21c1e269b8289d9471f9"
server_key_iv: "c68ffd2c0c186157812f564a7e5f9086"
server_key_version: "encryptionkey@0"
I20260620 14:34:48.349146 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "e5d207308ce0466ea94d5123137d1eea"
format_stamp: "Formatted at 2026-06-20 14:34:48 on dist-test-slave-gqzk"
server_key: "36dd68f8d02d21c1e269b8289d9471f9"
server_key_iv: "c68ffd2c0c186157812f564a7e5f9086"
server_key_version: "encryptionkey@0"
I20260620 14:34:48.352864 25431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:48.355307 25446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:48.356542 25431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:48.356653 25431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e5d207308ce0466ea94d5123137d1eea"
format_stamp: "Formatted at 2026-06-20 14:34:48 on dist-test-slave-gqzk"
server_key: "36dd68f8d02d21c1e269b8289d9471f9"
server_key_iv: "c68ffd2c0c186157812f564a7e5f9086"
server_key_version: "encryptionkey@0"
I20260620 14:34:48.356796 25431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:48.361353 25431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:48.364342 25431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:48.364514 25431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:48.372046 25431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44351
I20260620 14:34:48.372229 25498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44351 every 8 connection(s)
I20260620 14:34:48.373102 25431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:48.375847 25499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:48.377278 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25431
I20260620 14:34:48.377408 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:48.377698 4329 external_mini_cluster.cc:1428] Setting key 1cf742d2fa070bebc8439202b7be5bd3
I20260620 14:34:48.382128 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea: Bootstrap starting.
I20260620 14:34:48.382782 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:48.379128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:40051 (local address 127.0.0.1:44351)
0620 14:34:48.379571 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:48.379586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:48.379621 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:34:48.380246 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:48.380261 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:48.380553 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:48.380893 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:48.381636 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:48.381641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:48.381652 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:48.381662 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:48.381713 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:48.381732 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:48.381852 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:48.381921 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:34:48.382075 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
I20260620 14:34:48.384618 25499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:48.385492 25499 log.cc:824] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:48.385669 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:48.387531 25499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea: No bootstrap required, opened a new log
I20260620 14:34:48.389640 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:48.387775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:54773 (local address 127.0.0.1:44351)
0620 14:34:48.387883 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:48.387886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:48.388252 (+ 366us) server_negotiation.cc:407] Connection header received
0620 14:34:48.388369 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:48.388372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:48.388415 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:48.388493 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:48.389288 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:48.389292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:48.389295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:48.389308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:48.389337 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:48.389350 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:48.389441 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:48.389487 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:34:48.389525 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:34:48.391706 25499 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260620 14:34:48.391958 25499 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:48.392016 25499 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d207308ce0466ea94d5123137d1eea, State: Initialized, Role: FOLLOWER
I20260620 14:34:48.392498 25499 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260620 14:34:48.392640 25499 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:48.392696 25499 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:48.392803 25499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:48.393699 25499 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260620 14:34:48.393998 25499 leader_election.cc:302] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5d207308ce0466ea94d5123137d1eea; no voters:
I20260620 14:34:48.394254 25499 leader_election.cc:288] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:48.394400 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:48.394619 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [term 1 LEADER]: Becoming Leader. State: Replica: e5d207308ce0466ea94d5123137d1eea, State: Running, Role: LEADER
I20260620 14:34:48.394917 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } }
I20260620 14:34:48.395462 25499 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:48.396389 25511 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5d207308ce0466ea94d5123137d1eea. Latest consensus state: current_term: 1 leader_uuid: "e5d207308ce0466ea94d5123137d1eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } } }
I20260620 14:34:48.396514 25511 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:48.396776 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5d207308ce0466ea94d5123137d1eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5d207308ce0466ea94d5123137d1eea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44351 } } }
I20260620 14:34:48.396867 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:48.396993 25518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:48.399917 25518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:48.405921 25518 catalog_manager.cc:1347] Generated new cluster ID: 1151abd16b724dbf935946db897e7f02
I20260620 14:34:48.405988 25518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:48.414249 25518 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:48.414577 25518 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:48.419937 25518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5d207308ce0466ea94d5123137d1eea: Generated new TSK 0
I20260620 14:34:48.420637 25518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:48.434442 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25431
2026-06-20T14:34:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:48 dist-test-slave-gqzk krb5kdc[25530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:48 dist-test-slave-gqzk krb5kdc[25530](info): set up 2 sockets
Jun 20 14:34:48 dist-test-slave-gqzk krb5kdc[25530](info): commencing operation
krb5kdc: starting...
W20260620 14:34:49.384817 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:49.517846 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:49.628175 25543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:49.628463 25543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:49.628505 25543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:49.632669 25543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:49.632778 25543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:49.632800 25543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:49.632815 25543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:49.632830 25543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:49.638337 25543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:49.639436 25543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:49.640476 25543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:49.648952 25548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:49.648924 25549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:49.648952 25543 server_base.cc:1034] running on GCE node
W20260620 14:34:49.648974 25551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:49.649832 25543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:49.650961 25543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:49.652156 25543 hybrid_clock.cc:648] HybridClock initialized: now 1781966089652132 us; error 36 us; skew 500 ppm
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:49.655017 25543 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:49.655962 25543 webserver.cc:460] Webserver started at http://127.0.0.1:37719/ using document root <none> and password file <none>
I20260620 14:34:49.656502 25543 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:49.656569 25543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:49.656788 25543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:49.658876 25543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "6960f55697114b728af30e86053619bf"
format_stamp: "Formatted at 2026-06-20 14:34:49 on dist-test-slave-gqzk"
server_key: "cc1662c6f8f13705df9bf45f2eeaf581"
server_key_iv: "e4f60f3e028f6f372ed42548dc0f0f79"
server_key_version: "encryptionkey@0"
I20260620 14:34:49.659440 25543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "6960f55697114b728af30e86053619bf"
format_stamp: "Formatted at 2026-06-20 14:34:49 on dist-test-slave-gqzk"
server_key: "cc1662c6f8f13705df9bf45f2eeaf581"
server_key_iv: "e4f60f3e028f6f372ed42548dc0f0f79"
server_key_version: "encryptionkey@0"
I20260620 14:34:49.663329 25543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 14:34:49.665925 25558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:49.667220 25543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:49.667341 25543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "6960f55697114b728af30e86053619bf"
format_stamp: "Formatted at 2026-06-20 14:34:49 on dist-test-slave-gqzk"
server_key: "cc1662c6f8f13705df9bf45f2eeaf581"
server_key_iv: "e4f60f3e028f6f372ed42548dc0f0f79"
server_key_version: "encryptionkey@0"
I20260620 14:34:49.667449 25543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:49.682652 25543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:49.685762 25543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:49.685966 25543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:49.692924 25543 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43719
I20260620 14:34:49.692946 25610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43719 every 8 connection(s)
I20260620 14:34:49.693902 25543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:49.694161 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25543
I20260620 14:34:49.694269 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:49.694602 4329 external_mini_cluster.cc:1428] Setting key e63c48ecd2db1d2ff5b1de7504c0dfab
I20260620 14:34:49.697121 25611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:49.703915 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: Bootstrap starting.
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:34:49.706598 25611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:49.707450 25611 log.cc:824] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:49.709781 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: No bootstrap required, opened a new log
I20260620 14:34:49.710023 25614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:49.696055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:45259 (local address 127.0.0.1:43719)
0620 14:34:49.696483 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:49.696501 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:49.696543 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:34:49.697401 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:49.697423 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:49.697861 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:49.698248 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:49.698924 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:49.700456 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:49.701489 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:49.702196 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:49.702355 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:49.704903 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:49.704910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:49.704926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:49.704963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:49.706842 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:49.707333 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:49.707339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:49.707345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:49.707485 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:49.707889 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:49.707899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:49.707900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:49.708328 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:49.708540 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:49.708859 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:34:49.709079 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260620 14:34:49.713392 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:49.713440 25611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260620 14:34:49.713689 25611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:49.713748 25611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6960f55697114b728af30e86053619bf, State: Initialized, Role: FOLLOWER
I20260620 14:34:49.714262 25611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260620 14:34:49.714396 25611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:49.714452 25611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:49.714607 25611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:49.715636 25611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260620 14:34:49.716013 25611 leader_election.cc:302] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6960f55697114b728af30e86053619bf; no voters:
I20260620 14:34:49.716342 25611 leader_election.cc:288] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:49.716434 25621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:49.716660 25621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [term 1 LEADER]: Becoming Leader. State: Replica: 6960f55697114b728af30e86053619bf, State: Running, Role: LEADER
I20260620 14:34:49.717017 25621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } }
I20260620 14:34:49.717693 25611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:49.720026 25622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6960f55697114b728af30e86053619bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } } }
I20260620 14:34:49.720155 25622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:49.720265 25623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6960f55697114b728af30e86053619bf. Latest consensus state: current_term: 1 leader_uuid: "6960f55697114b728af30e86053619bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6960f55697114b728af30e86053619bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43719 } } }
I20260620 14:34:49.720464 25623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf [sys.catalog]: This master's current role is: LEADER
W20260620 14:34:49.723129 25635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:49.723268 25635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:49.723402 25637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:49.724371 25614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:49.715361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:55413 (local address 127.0.0.1:43719)
0620 14:34:49.715478 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:49.715483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:49.715585 (+ 102us) server_negotiation.cc:407] Connection header received
0620 14:34:49.715669 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:49.715671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:49.715731 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:49.715805 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:49.716567 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:49.717358 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:49.718458 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:49.718796 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:49.718888 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:49.720849 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:49.720851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:49.720854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:49.720876 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:49.722397 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:49.722966 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:49.722970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:49.722972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:49.723060 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:49.723423 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:49.723426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:49.723428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:49.723639 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:49.723799 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:49.724122 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:34:49.724222 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:34:49.724223 25637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:49.730180 25637 catalog_manager.cc:1347] Generated new cluster ID: abaccd392d804dbbbdbd50a1ae491106
I20260620 14:34:49.730250 25637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:49.750919 25637 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:49.751959 25637 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:49.772050 25637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6960f55697114b728af30e86053619bf: Generated new TSK 0
I20260620 14:34:49.772893 25637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:49.777616 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25543
W20260620 14:34:49.782639 25528 connection.cc:535] client connection to 127.0.0.1:43719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43719): Connection reset by peer (error 104)
2026-06-20T14:34:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25642](info): set up 2 sockets
Jun 20 14:34:49 dist-test-slave-gqzk krb5kdc[25642](info): commencing operation
krb5kdc: starting...
W20260620 14:34:50.732775 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:50 dist-test-slave-gqzk krb5kdc[25642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:50.852417 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:50.972083 25655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:50.972362 25655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:50.972409 25655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:50.976473 25655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:50.976543 25655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:50.976562 25655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:50.976577 25655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:50.976591 25655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:50.981958 25655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:50.983013 25655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:50.984043 25655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:50.993021 25661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:50.993070 25663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:50.993052 25660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:50.993220 25655 server_base.cc:1034] running on GCE node
I20260620 14:34:50.994010 25655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:50.995187 25655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:50.996392 25655 hybrid_clock.cc:648] HybridClock initialized: now 1781966090996369 us; error 34 us; skew 500 ppm
Jun 20 14:34:50 dist-test-slave-gqzk krb5kdc[25642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:50.999509 25655 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:51.000407 25655 webserver.cc:460] Webserver started at http://127.0.0.1:42357/ using document root <none> and password file <none>
I20260620 14:34:51.000994 25655 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:51.001083 25655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:51.001305 25655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:51.003355 25655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "6ba19758491d413d9778c22e3f4e450c"
format_stamp: "Formatted at 2026-06-20 14:34:51 on dist-test-slave-gqzk"
server_key: "6c48f71556e6a5e88be46755e9adec1b"
server_key_iv: "bccdb7ebab2a879693b731c0bfb2539a"
server_key_version: "encryptionkey@0"
I20260620 14:34:51.003897 25655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "6ba19758491d413d9778c22e3f4e450c"
format_stamp: "Formatted at 2026-06-20 14:34:51 on dist-test-slave-gqzk"
server_key: "6c48f71556e6a5e88be46755e9adec1b"
server_key_iv: "bccdb7ebab2a879693b731c0bfb2539a"
server_key_version: "encryptionkey@0"
I20260620 14:34:51.007788 25655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:34:51.010502 25670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:51.011942 25655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 14:34:51.012084 25655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "6ba19758491d413d9778c22e3f4e450c"
format_stamp: "Formatted at 2026-06-20 14:34:51 on dist-test-slave-gqzk"
server_key: "6c48f71556e6a5e88be46755e9adec1b"
server_key_iv: "bccdb7ebab2a879693b731c0bfb2539a"
server_key_version: "encryptionkey@0"
I20260620 14:34:51.012197 25655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:51.027778 25655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:51.030929 25655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:51.031128 25655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:51.038221 25655 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43573
I20260620 14:34:51.038229 25722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43573 every 8 connection(s)
I20260620 14:34:51.039211 25655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:51.039462 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25655
I20260620 14:34:51.039580 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:51.039887 4329 external_mini_cluster.cc:1428] Setting key 4662dd3f7ccc8fc2a1ce4d7fc387c631
I20260620 14:34:51.042392 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:51.048080 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: Bootstrap starting.
I20260620 14:34:51.048873 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:51.041366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:50235 (local address 127.0.0.1:43573)
0620 14:34:51.041829 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:51.041845 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:51.041882 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:34:51.042564 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:51.042583 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:51.042950 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:51.043274 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:51.043687 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:51.044680 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:51.045627 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:51.046114 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:51.046260 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:51.047157 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:51.047162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:51.047175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:51.047563 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:51.047616 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:51.047639 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:51.047757 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:51.047941 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:34:51.048079 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260620 14:34:51.050402 25723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:51.051231 25723 log.cc:824] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:51.051738 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:51.053213 25723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: No bootstrap required, opened a new log
I20260620 14:34:51.057706 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:51.053719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:55195 (local address 127.0.0.1:43573)
0620 14:34:51.053826 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:51.053829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:51.054011 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:34:51.054131 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:51.054133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:51.054173 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:51.054237 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:51.054527 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:51.055038 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:51.056140 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:51.056350 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:51.056418 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:51.057268 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:51.057271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:51.057273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:51.057289 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:51.057315 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:51.057325 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:51.057389 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:51.057531 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:34:51.057568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:34:51.057672 25723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } }
I20260620 14:34:51.058004 25723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:51.058063 25723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba19758491d413d9778c22e3f4e450c, State: Initialized, Role: FOLLOWER
I20260620 14:34:51.058491 25723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } }
I20260620 14:34:51.058622 25723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:51.058691 25723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:51.058802 25723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:51.059746 25723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } }
I20260620 14:34:51.060058 25723 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba19758491d413d9778c22e3f4e450c; no voters:
I20260620 14:34:51.060322 25723 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:51.060421 25733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:51.060609 25733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [term 1 LEADER]: Becoming Leader. State: Replica: 6ba19758491d413d9778c22e3f4e450c, State: Running, Role: LEADER
I20260620 14:34:51.060971 25733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } }
I20260620 14:34:51.061558 25723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:51.062616 25734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba19758491d413d9778c22e3f4e450c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } } }
I20260620 14:34:51.062747 25734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:51.063835 25735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba19758491d413d9778c22e3f4e450c. Latest consensus state: current_term: 1 leader_uuid: "6ba19758491d413d9778c22e3f4e450c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba19758491d413d9778c22e3f4e450c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43573 } } }
I20260620 14:34:51.063930 25735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c [sys.catalog]: This master's current role is: LEADER
W20260620 14:34:51.066421 25748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:51.066504 25748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:51.066634 25742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:51.067373 25742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:51.072542 25742 catalog_manager.cc:1347] Generated new cluster ID: 1e2932cb8395410e8012a20586168b82
I20260620 14:34:51.072614 25742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:51.094770 25742 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:51.095676 25742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:51.100343 25742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba19758491d413d9778c22e3f4e450c: Generated new TSK 0
I20260620 14:34:51.101119 25742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:51.113114 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25655
W20260620 14:34:51.118085 25640 connection.cc:535] client connection to 127.0.0.1:43573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43573): Connection reset by peer (error 104)
2026-06-20T14:34:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:51 dist-test-slave-gqzk krb5kdc[25754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:51 dist-test-slave-gqzk krb5kdc[25754](info): set up 2 sockets
Jun 20 14:34:51 dist-test-slave-gqzk krb5kdc[25754](info): commencing operation
krb5kdc: starting...
W20260620 14:34:52.072731 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:52 dist-test-slave-gqzk krb5kdc[25754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:34:52.192108 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:52.300554 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:52.300875 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:52.300938 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:52.304991 25767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:52.305060 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:52.305089 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:52.305109 25767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:52.305130 25767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:52.310544 25767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:52.311625 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:52.312647 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:52.321044 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.321070 25773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:52.321056 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:52.321275 25767 server_base.cc:1034] running on GCE node
I20260620 14:34:52.321832 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:52.323042 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:52.324226 25767 hybrid_clock.cc:648] HybridClock initialized: now 1781966092324206 us; error 35 us; skew 500 ppm
Jun 20 14:34:52 dist-test-slave-gqzk krb5kdc[25754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:52.331149 25767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:34:52.332227 25767 webserver.cc:460] Webserver started at http://127.0.0.1:42157/ using document root <none> and password file <none>
I20260620 14:34:52.332862 25767 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:52.332943 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:52.333158 25767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:52.335373 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "7e043cad51da43deb8933cd4cdb8381b"
format_stamp: "Formatted at 2026-06-20 14:34:52 on dist-test-slave-gqzk"
server_key: "9dfff88eb05a821cefcd711d164971e0"
server_key_iv: "9e20909dcf8384ce51a34d5ce7e708c7"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.335959 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "7e043cad51da43deb8933cd4cdb8381b"
format_stamp: "Formatted at 2026-06-20 14:34:52 on dist-test-slave-gqzk"
server_key: "9dfff88eb05a821cefcd711d164971e0"
server_key_iv: "9e20909dcf8384ce51a34d5ce7e708c7"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.340003 25767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 14:34:52.342509 25782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:52.343771 25767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:34:52.343891 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "7e043cad51da43deb8933cd4cdb8381b"
format_stamp: "Formatted at 2026-06-20 14:34:52 on dist-test-slave-gqzk"
server_key: "9dfff88eb05a821cefcd711d164971e0"
server_key_iv: "9e20909dcf8384ce51a34d5ce7e708c7"
server_key_version: "encryptionkey@0"
I20260620 14:34:52.343997 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:52.348600 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:52.351737 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:52.351923 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:52.358655 25767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44561
I20260620 14:34:52.358692 25834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44561 every 8 connection(s)
I20260620 14:34:52.359568 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:52.362360 25835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:52.367931 25835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: Bootstrap starting.
I20260620 14:34:52.368386 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25767
I20260620 14:34:52.368489 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:52.368800 4329 external_mini_cluster.cc:1428] Setting key b7d5d2a49a70a836c5e75b373c635bca
I20260620 14:34:52.370630 25835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:52.371541 25835 log.cc:824] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:52.373909 25835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: No bootstrap required, opened a new log
I20260620 14:34:52.374608 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.370191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:50509 (local address 127.0.0.1:44561)
0620 14:34:52.370662 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:52.370682 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:52.370729 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:34:52.371620 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:52.371646 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:52.372028 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:52.372356 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:52.373097 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:52.373102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:52.373117 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:52.373130 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:52.373193 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:52.373220 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:52.373345 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:52.373472 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:34:52.373632 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
I20260620 14:34:52.377568 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:52.378193 25835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260620 14:34:52.378480 25835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:52.378536 25835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e043cad51da43deb8933cd4cdb8381b, State: Initialized, Role: FOLLOWER
I20260620 14:34:52.379029 25835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260620 14:34:52.379197 25835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:52.379266 25835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:52.379364 25835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:52.381290 25835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260620 14:34:52.381701 25835 leader_election.cc:302] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e043cad51da43deb8933cd4cdb8381b; no voters:
I20260620 14:34:52.381989 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:52.379863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.39:43803 (local address 127.0.0.1:44561)
0620 14:34:52.380028 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:52.380032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:52.380192 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:34:52.380355 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:52.380358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:52.380397 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:52.380472 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:52.381381 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:52.381384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:52.381387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:52.381399 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:52.381430 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:52.381443 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:52.381518 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:52.381797 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:34:52.381840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:34:52.382083 25835 leader_election.cc:288] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:52.382439 25846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:52.382637 25846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [term 1 LEADER]: Becoming Leader. State: Replica: 7e043cad51da43deb8933cd4cdb8381b, State: Running, Role: LEADER
I20260620 14:34:52.382970 25846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260620 14:34:52.383481 25835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:52.384502 25847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e043cad51da43deb8933cd4cdb8381b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260620 14:34:52.384647 25847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:52.385228 25846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e043cad51da43deb8933cd4cdb8381b. Latest consensus state: current_term: 1 leader_uuid: "7e043cad51da43deb8933cd4cdb8381b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e043cad51da43deb8933cd4cdb8381b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260620 14:34:52.385319 25846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b [sys.catalog]: This master's current role is: LEADER
W20260620 14:34:52.388509 25860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:52.388581 25860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:52.388677 25854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:52.389611 25854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:52.395383 25854 catalog_manager.cc:1347] Generated new cluster ID: 3cd3fd32df2d4e3fab4c4912b718d0bf
I20260620 14:34:52.395467 25854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:52.409956 25854 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:52.410317 25854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:52.422931 25854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e043cad51da43deb8933cd4cdb8381b: Generated new TSK 0
I20260620 14:34:52.423892 25854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:52.434183 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25767
2026-06-20T14:34:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:52 dist-test-slave-gqzk krb5kdc[25866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:52 dist-test-slave-gqzk krb5kdc[25866](info): set up 2 sockets
Jun 20 14:34:52 dist-test-slave-gqzk krb5kdc[25866](info): commencing operation
krb5kdc: starting...
W20260620 14:34:53.389580 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:53.511373 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40647
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:53.622361 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:53.622650 25879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:53.622692 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:53.626811 25879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:53.626888 25879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:53.626906 25879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:53.626922 25879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:53.626937 25879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:53.632426 25879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:40647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:53.633553 25879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:53.634564 25879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:53.643185 25885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:53.643194 25884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:53.643194 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:53.643205 25879 server_base.cc:1034] running on GCE node
I20260620 14:34:53.644007 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:53.645200 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:53.646407 25879 hybrid_clock.cc:648] HybridClock initialized: now 1781966093646379 us; error 46 us; skew 500 ppm
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:53.649156 25879 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:53.650123 25879 webserver.cc:460] Webserver started at http://127.4.58.126:44341/ using document root <none> and password file <none>
I20260620 14:34:53.650676 25879 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:53.650746 25879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:53.650938 25879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:53.653088 25879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "e0548f1a672642359e0e142257ed7334"
format_stamp: "Formatted at 2026-06-20 14:34:53 on dist-test-slave-gqzk"
server_key: "1bec0cf6b352c352f8f10e5043702023"
server_key_iv: "59e6b85087fdadc6842924ea3239efc8"
server_key_version: "encryptionkey@0"
I20260620 14:34:53.653647 25879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "e0548f1a672642359e0e142257ed7334"
format_stamp: "Formatted at 2026-06-20 14:34:53 on dist-test-slave-gqzk"
server_key: "1bec0cf6b352c352f8f10e5043702023"
server_key_iv: "59e6b85087fdadc6842924ea3239efc8"
server_key_version: "encryptionkey@0"
I20260620 14:34:53.657397 25879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:34:53.660221 25894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:53.661623 25879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:34:53.661767 25879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "e0548f1a672642359e0e142257ed7334"
format_stamp: "Formatted at 2026-06-20 14:34:53 on dist-test-slave-gqzk"
server_key: "1bec0cf6b352c352f8f10e5043702023"
server_key_iv: "59e6b85087fdadc6842924ea3239efc8"
server_key_version: "encryptionkey@0"
I20260620 14:34:53.661878 25879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:53.676327 25879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:53.679579 25879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:53.679878 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:53.686811 25879 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:40647
I20260620 14:34:53.686830 25946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:40647 every 8 connection(s)
I20260620 14:34:53.687791 25879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:53.690537 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:53.695942 25947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334: Bootstrap starting.
I20260620 14:34:53.697845 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25879
I20260620 14:34:53.698010 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:53.698300 4329 external_mini_cluster.cc:1428] Setting key 31c626dc9978e978d2db247a695a0a09
I20260620 14:34:53.698683 25947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:53.699549 25947 log.cc:824] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:53.701596 25947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334: No bootstrap required, opened a new log
I20260620 14:34:53.705557 25947 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } }
I20260620 14:34:53.705822 25947 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:53.705875 25947 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0548f1a672642359e0e142257ed7334, State: Initialized, Role: FOLLOWER
I20260620 14:34:53.706426 25947 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } }
I20260620 14:34:53.706610 25947 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:53.706678 25947 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:53.706807 25947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:53.708073 25947 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } }
I20260620 14:34:53.708412 25947 leader_election.cc:302] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0548f1a672642359e0e142257ed7334; no voters:
I20260620 14:34:53.708709 25947 leader_election.cc:288] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:53.708854 25952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:53.709110 25952 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [term 1 LEADER]: Becoming Leader. State: Replica: e0548f1a672642359e0e142257ed7334, State: Running, Role: LEADER
I20260620 14:34:53.709458 25952 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } }
I20260620 14:34:53.710229 25947 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:53.711009 25953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0548f1a672642359e0e142257ed7334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } } }
I20260620 14:34:53.711190 25953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:53.711546 25954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0548f1a672642359e0e142257ed7334. Latest consensus state: current_term: 1 leader_uuid: "e0548f1a672642359e0e142257ed7334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0548f1a672642359e0e142257ed7334" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40647 } } }
I20260620 14:34:53.711695 25954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:53.712312 25959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:53.713876 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.699432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36173 (local address 127.4.58.126:40647)
0620 14:34:53.699830 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:53.699846 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:53.699890 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:34:53.700644 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:53.700665 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:53.701055 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:53.701357 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:53.702306 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.703286 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.704499 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.705038 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.705158 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.708101 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:53.708106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:53.708123 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:53.708155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:53.710260 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.710899 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.710905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.710912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.711047 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.711455 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.711459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.711461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.712002 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:53.712213 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:53.712886 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 14:34:53.713101 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260620 14:34:53.715268 25959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:53.717265 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:53.720701 25959 catalog_manager.cc:1347] Generated new cluster ID: c1bdebd7eb4644e792f811b705c559cf
I20260620 14:34:53.720786 25959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:53.726786 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:53.719427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34775 (local address 127.4.58.126:40647)
0620 14:34:53.719525 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:53.719529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:53.719833 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:34:53.719951 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:53.719954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:53.720003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:53.720083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:53.720881 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.721412 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.722183 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:53.722454 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:53.722542 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:53.723847 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:53.723851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:53.723854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:53.723876 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:53.725178 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.725620 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.725623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.725625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.725693 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:53.726001 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:53.726005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:53.726007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:53.726165 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:53.726288 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:53.726551 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:34:53.726636 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:34:53.737905 25959 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:53.738862 25959 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:53.747421 25959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0548f1a672642359e0e142257ed7334: Generated new TSK 0
I20260620 14:34:53.748351 25959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:53.760243 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25879
W20260620 14:34:53.764977 25864 connection.cc:535] client connection to 127.4.58.126:40647 recv error: Network error: failed to read from TLS socket (remote: 127.4.58.126:40647): Connection reset by peer (error 104)
2026-06-20T14:34:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25978](info): set up 2 sockets
Jun 20 14:34:53 dist-test-slave-gqzk krb5kdc[25978](info): commencing operation
krb5kdc: starting...
W20260620 14:34:54.724788 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:54 dist-test-slave-gqzk krb5kdc[25978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:54.843917 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34223
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:33623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:54.951696 25991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:54.951980 25991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:54.952023 25991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:54.956029 25991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:54.956089 25991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:54.956107 25991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:54.956123 25991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:54.956138 25991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:54.961555 25991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:33623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:54.962603 25991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:54.963642 25991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:54.971999 25997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:54.972101 25991 server_base.cc:1034] running on GCE node
W20260620 14:34:54.972048 25996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:54.972034 25999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:54.972839 25991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:54.973946 25991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:54.975098 25991 hybrid_clock.cc:648] HybridClock initialized: now 1781966094975058 us; error 55 us; skew 500 ppm
Jun 20 14:34:54 dist-test-slave-gqzk krb5kdc[25978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966094, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:54.977847 25991 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:54.978710 25991 webserver.cc:460] Webserver started at http://127.4.58.126:35339/ using document root <none> and password file <none>
I20260620 14:34:54.979246 25991 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:54.979307 25991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:54.979530 25991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:54.981611 25991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a"
format_stamp: "Formatted at 2026-06-20 14:34:54 on dist-test-slave-gqzk"
server_key: "be764fdf2bd4033ad58811178f0c7b69"
server_key_iv: "7189c462568e8cc13b158385c18ad0d0"
server_key_version: "encryptionkey@0"
I20260620 14:34:54.982136 25991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a"
format_stamp: "Formatted at 2026-06-20 14:34:54 on dist-test-slave-gqzk"
server_key: "be764fdf2bd4033ad58811178f0c7b69"
server_key_iv: "7189c462568e8cc13b158385c18ad0d0"
server_key_version: "encryptionkey@0"
I20260620 14:34:54.985816 25991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:54.988291 26006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:54.989653 25991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:54.989766 25991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a"
format_stamp: "Formatted at 2026-06-20 14:34:54 on dist-test-slave-gqzk"
server_key: "be764fdf2bd4033ad58811178f0c7b69"
server_key_iv: "7189c462568e8cc13b158385c18ad0d0"
server_key_version: "encryptionkey@0"
I20260620 14:34:54.989887 25991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:55.023028 25991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:55.026618 25991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:55.026814 25991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:55.033697 25991 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34223
I20260620 14:34:55.033718 26058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34223 every 8 connection(s)
I20260620 14:34:55.034634 25991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:55.037473 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:55.040542 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 25991
I20260620 14:34:55.040670 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:55.040995 4329 external_mini_cluster.cc:1428] Setting key 945c65f501fe2910ffa23b3da5265143
I20260620 14:34:55.043484 26059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: Bootstrap starting.
I20260620 14:34:55.045976 26059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:55.046896 26059 log.cc:824] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:55.049288 26059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: No bootstrap required, opened a new log
I20260620 14:34:55.050388 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:55.042420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44373 (local address 127.4.58.126:34223)
0620 14:34:55.042812 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:55.042827 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:55.042870 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:34:55.043501 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:55.043518 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:55.043916 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:55.044206 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:55.044584 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:55.045728 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:55.046984 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:55.047580 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:55.047734 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:55.048645 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:55.048650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:55.048664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:55.049105 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:55.049158 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:55.049181 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:55.049280 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:55.049493 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:34:55.049636 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260620 14:34:55.053013 26059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } }
I20260620 14:34:55.053223 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:55.053287 26059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:55.053339 26059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbd93ea0bf649b19ac46ed4dfa8699a, State: Initialized, Role: FOLLOWER
I20260620 14:34:55.053802 26059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } }
I20260620 14:34:55.053958 26059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:55.054031 26059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:55.054133 26059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:55.055315 26059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } }
I20260620 14:34:55.055685 26059 leader_election.cc:302] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cbd93ea0bf649b19ac46ed4dfa8699a; no voters:
I20260620 14:34:55.056016 26059 leader_election.cc:288] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:55.056106 26069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:55.056389 26069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [term 1 LEADER]: Becoming Leader. State: Replica: 2cbd93ea0bf649b19ac46ed4dfa8699a, State: Running, Role: LEADER
I20260620 14:34:55.056797 26069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } }
I20260620 14:34:55.057446 26059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:55.058555 26070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } } }
I20260620 14:34:55.058677 26070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:55.059082 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cbd93ea0bf649b19ac46ed4dfa8699a. Latest consensus state: current_term: 1 leader_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cbd93ea0bf649b19ac46ed4dfa8699a" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34223 } } }
I20260620 14:34:55.059949 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:55.060236 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:55.055392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50875 (local address 127.4.58.126:34223)
0620 14:34:55.055502 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:55.055506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:55.055710 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:34:55.055893 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:55.055895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:55.055936 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:55.055996 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:55.056310 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:55.057060 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:55.058272 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:55.058636 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:55.058736 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:55.059769 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:55.059773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:55.059775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:55.059787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:55.059814 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:55.059826 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:55.059893 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:55.060067 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:34:55.060105 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:34:55.063117 26084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:34:55.063199 26084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:34:55.063292 26078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:55.064030 26078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:55.069140 26078 catalog_manager.cc:1347] Generated new cluster ID: c6a859ea7fde4808aeda47c533253c32
I20260620 14:34:55.069211 26078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:55.086658 26078 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:55.088011 26078 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:55.101918 26078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cbd93ea0bf649b19ac46ed4dfa8699a: Generated new TSK 0
I20260620 14:34:55.102638 26078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:55.116061 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 25991
W20260620 14:34:55.120932 25976 connection.cc:535] client connection to 127.4.58.126:34223 recv error: Network error: failed to read from TLS socket (remote: 127.4.58.126:34223): Connection reset by peer (error 104)
2026-06-20T14:34:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:55 dist-test-slave-gqzk krb5kdc[26090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:55 dist-test-slave-gqzk krb5kdc[26090](info): set up 2 sockets
Jun 20 14:34:55 dist-test-slave-gqzk krb5kdc[26090](info): commencing operation
krb5kdc: starting...
W20260620 14:34:56.076802 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:56 dist-test-slave-gqzk krb5kdc[26090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:56.199569 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38581
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:56.309280 26103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:56.309574 26103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:56.309633 26103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:56.313723 26103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:56.313802 26103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:56.313827 26103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:56.313858 26103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:56.313886 26103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:56.319407 26103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:38581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:56.320544 26103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:56.321632 26103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:56.330327 26111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:56.330329 26108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:56.330327 26109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:56.330629 26103 server_base.cc:1034] running on GCE node
I20260620 14:34:56.331383 26103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:56.332545 26103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:56.333742 26103 hybrid_clock.cc:648] HybridClock initialized: now 1781966096333712 us; error 42 us; skew 500 ppm
Jun 20 14:34:56 dist-test-slave-gqzk krb5kdc[26090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:56.336659 26103 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:56.337743 26103 webserver.cc:460] Webserver started at http://127.4.58.126:38363/ using document root <none> and password file <none>
I20260620 14:34:56.338339 26103 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:56.338414 26103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:56.338617 26103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:56.340804 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "57862949e7e34632bf9ab3b377d1f668"
format_stamp: "Formatted at 2026-06-20 14:34:56 on dist-test-slave-gqzk"
server_key: "e1cb61ae312c4fc64591df87b3119639"
server_key_iv: "4184f0953c4eb39f9fc792e03cabe2d4"
server_key_version: "encryptionkey@0"
I20260620 14:34:56.341420 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "57862949e7e34632bf9ab3b377d1f668"
format_stamp: "Formatted at 2026-06-20 14:34:56 on dist-test-slave-gqzk"
server_key: "e1cb61ae312c4fc64591df87b3119639"
server_key_iv: "4184f0953c4eb39f9fc792e03cabe2d4"
server_key_version: "encryptionkey@0"
I20260620 14:34:56.345585 26103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:34:56.348090 26118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:56.349417 26103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:34:56.349551 26103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "57862949e7e34632bf9ab3b377d1f668"
format_stamp: "Formatted at 2026-06-20 14:34:56 on dist-test-slave-gqzk"
server_key: "e1cb61ae312c4fc64591df87b3119639"
server_key_iv: "4184f0953c4eb39f9fc792e03cabe2d4"
server_key_version: "encryptionkey@0"
I20260620 14:34:56.349659 26103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:56.354543 26103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:56.357573 26103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:56.357849 26103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:56.364678 26103 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:38581
I20260620 14:34:56.364705 26170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:38581 every 8 connection(s)
I20260620 14:34:56.365662 26103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:56.368446 26171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:56.374110 26171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668: Bootstrap starting.
I20260620 14:34:56.375751 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26103
I20260620 14:34:56.375905 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:56.376160 4329 external_mini_cluster.cc:1428] Setting key cbe14b841b0665ec6fbbf5ad993bbc13
I20260620 14:34:56.376654 26171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:56.377544 26171 log.cc:824] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:56.379603 26171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668: No bootstrap required, opened a new log
I20260620 14:34:56.381347 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:56.377340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42913 (local address 127.4.58.126:38581)
0620 14:34:56.377787 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:56.377801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:56.377837 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:34:56.378630 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:56.378650 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:56.378950 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:56.379240 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:56.380051 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:56.380056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:56.380068 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:56.380077 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:56.380127 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:56.380145 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:56.380232 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:56.380402 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:34:56.380575 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260620 14:34:56.383425 26171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } }
I20260620 14:34:56.383666 26171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:56.383729 26171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57862949e7e34632bf9ab3b377d1f668, State: Initialized, Role: FOLLOWER
I20260620 14:34:56.384178 26171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } }
I20260620 14:34:56.384274 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:56.384325 26171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:56.384392 26171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:56.384480 26171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:56.385627 26171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } }
I20260620 14:34:56.386055 26171 leader_election.cc:302] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57862949e7e34632bf9ab3b377d1f668; no voters:
I20260620 14:34:56.386391 26171 leader_election.cc:288] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:56.386478 26180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:56.386660 26180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [term 1 LEADER]: Becoming Leader. State: Replica: 57862949e7e34632bf9ab3b377d1f668, State: Running, Role: LEADER
I20260620 14:34:56.387019 26180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } }
I20260620 14:34:56.387715 26171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:56.388204 26180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57862949e7e34632bf9ab3b377d1f668. Latest consensus state: current_term: 1 leader_uuid: "57862949e7e34632bf9ab3b377d1f668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } } }
I20260620 14:34:56.388624 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:56.386902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34663 (local address 127.4.58.126:38581)
0620 14:34:56.387026 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:56.387031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:56.387156 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:34:56.387327 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:56.387330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:56.387370 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:56.387435 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:56.388141 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:56.388145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:56.388148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:56.388160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:56.388189 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:56.388202 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:56.388254 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:56.388471 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:34:56.388512 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:34:56.388831 26181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57862949e7e34632bf9ab3b377d1f668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57862949e7e34632bf9ab3b377d1f668" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 38581 } } }
I20260620 14:34:56.389190 26180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:56.390094 26181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668 [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:56.390110 26193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:56.392724 26193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:56.398352 26193 catalog_manager.cc:1347] Generated new cluster ID: f9d306fa7439460ab6537b83a8ea4145
I20260620 14:34:56.398480 26193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:56.416574 26193 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:56.417021 26193 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:56.427202 26193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57862949e7e34632bf9ab3b377d1f668: Generated new TSK 0
I20260620 14:34:56.428117 26193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:56.437786 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26103
2026-06-20T14:34:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:56 dist-test-slave-gqzk krb5kdc[26202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:56 dist-test-slave-gqzk krb5kdc[26202](info): set up 2 sockets
Jun 20 14:34:56 dist-test-slave-gqzk krb5kdc[26202](info): commencing operation
krb5kdc: starting...
W20260620 14:34:57.388693 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:57.508086 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40697
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:57.617944 26215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:57.618227 26215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:57.618269 26215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:57.622366 26215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:57.622437 26215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:57.622454 26215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:57.622470 26215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:57.622484 26215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:57.628000 26215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:57.629107 26215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:57.630112 26215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:57.638552 26223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:57.638598 26215 server_base.cc:1034] running on GCE node
W20260620 14:34:57.638552 26221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:57.638577 26220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:57.639334 26215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:57.640450 26215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:57.641606 26215 hybrid_clock.cc:648] HybridClock initialized: now 1781966097641573 us; error 47 us; skew 500 ppm
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:57.648823 26215 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:57.649748 26215 webserver.cc:460] Webserver started at http://127.4.58.126:38075/ using document root <none> and password file <none>
I20260620 14:34:57.650305 26215 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:57.650377 26215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:57.650593 26215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:57.652650 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "d3360a64da3645718c0e135aa2cb0acf"
format_stamp: "Formatted at 2026-06-20 14:34:57 on dist-test-slave-gqzk"
server_key: "44b38f4ee13d21429ebf4fcf82d32bff"
server_key_iv: "3ac5a649de0c9ad5d38b1dc92a02ca9c"
server_key_version: "encryptionkey@0"
I20260620 14:34:57.653257 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "d3360a64da3645718c0e135aa2cb0acf"
format_stamp: "Formatted at 2026-06-20 14:34:57 on dist-test-slave-gqzk"
server_key: "44b38f4ee13d21429ebf4fcf82d32bff"
server_key_iv: "3ac5a649de0c9ad5d38b1dc92a02ca9c"
server_key_version: "encryptionkey@0"
I20260620 14:34:57.656879 26215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:34:57.659221 26230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:57.660466 26215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:34:57.660576 26215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "d3360a64da3645718c0e135aa2cb0acf"
format_stamp: "Formatted at 2026-06-20 14:34:57 on dist-test-slave-gqzk"
server_key: "44b38f4ee13d21429ebf4fcf82d32bff"
server_key_iv: "3ac5a649de0c9ad5d38b1dc92a02ca9c"
server_key_version: "encryptionkey@0"
I20260620 14:34:57.660668 26215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:57.673529 26215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:57.676956 26215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:57.677140 26215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:57.683948 26215 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:40697
I20260620 14:34:57.683990 26282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:40697 every 8 connection(s)
I20260620 14:34:57.684957 26215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:57.687773 26283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:57.693398 26283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf: Bootstrap starting.
I20260620 14:34:57.694423 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26215
I20260620 14:34:57.694554 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:57.694788 4329 external_mini_cluster.cc:1428] Setting key 6e99a564cb170b68b49565e5a8f901d5
I20260620 14:34:57.696153 26283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:57.697086 26283 log.cc:824] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:57.699246 26283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf: No bootstrap required, opened a new log
I20260620 14:34:57.703388 26283 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } }
I20260620 14:34:57.703676 26283 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:57.703747 26283 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3360a64da3645718c0e135aa2cb0acf, State: Initialized, Role: FOLLOWER
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:34:57.704255 26283 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } }
I20260620 14:34:57.704386 26283 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:57.704449 26283 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:57.704545 26283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:57.705716 26283 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } }
I20260620 14:34:57.706115 26283 leader_election.cc:302] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3360a64da3645718c0e135aa2cb0acf; no voters:
I20260620 14:34:57.706487 26283 leader_election.cc:288] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:57.706570 26288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:57.706784 26288 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [term 1 LEADER]: Becoming Leader. State: Replica: d3360a64da3645718c0e135aa2cb0acf, State: Running, Role: LEADER
I20260620 14:34:57.707125 26288 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } }
I20260620 14:34:57.707933 26283 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:57.708671 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3360a64da3645718c0e135aa2cb0acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } } }
I20260620 14:34:57.708885 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:57.709153 26289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3360a64da3645718c0e135aa2cb0acf. Latest consensus state: current_term: 1 leader_uuid: "d3360a64da3645718c0e135aa2cb0acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3360a64da3645718c0e135aa2cb0acf" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 40697 } } }
I20260620 14:34:57.709282 26289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:57.710305 26297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:57.710834 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:57.695898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52071 (local address 127.4.58.126:40697)
0620 14:34:57.696370 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:57.696391 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:57.696438 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:34:57.697330 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:57.697354 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:57.697737 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:57.698124 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:57.698755 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:57.700209 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:57.701084 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:57.701821 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:57.702005 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:57.704771 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:57.704779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:57.704797 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:57.704833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:57.706769 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:57.707318 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:57.707324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:57.707330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:57.707471 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:57.707783 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:57.707787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:57.707789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:57.708339 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:57.709332 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:57.709495 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:34:57.709830 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":125,"threads_started":1}
I20260620 14:34:57.713778 26297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:57.714128 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:57.722301 26297 catalog_manager.cc:1347] Generated new cluster ID: 6539b71a13ee4bc2a55e281e52a1b719
I20260620 14:34:57.722395 26297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:57.727077 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:57.716251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36507 (local address 127.4.58.126:40697)
0620 14:34:57.716366 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:57.716370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:57.716595 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:34:57.716702 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:57.716705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:57.716777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:57.716848 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:57.717483 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:57.718067 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:57.719248 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:57.719483 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:57.719561 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:57.721196 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:57.721199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:57.721202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:34:57.721227 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:57.725478 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:57.725969 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:57.725972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:57.725974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:57.726049 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:34:57.726286 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:34:57.726288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:34:57.726290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:34:57.726474 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:57.726592 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:57.726802 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:34:57.726929 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:34:57.733815 26297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:57.734824 26297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:57.746384 26297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3360a64da3645718c0e135aa2cb0acf: Generated new TSK 0
I20260620 14:34:57.747184 26297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:57.750828 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26215
2026-06-20T14:34:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26314](info): set up 2 sockets
Jun 20 14:34:57 dist-test-slave-gqzk krb5kdc[26314](info): commencing operation
krb5kdc: starting...
W20260620 14:34:58.721020 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:34:58 dist-test-slave-gqzk krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:34:58.839838 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42069
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:34:58.948334 26327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:34:58.948624 26327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:34:58.948683 26327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:34:58.952855 26327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:34:58.952934 26327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:34:58.952960 26327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:34:58.952979 26327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:34:58.952998 26327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:34:58.958433 26327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:42069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:34:58.959511 26327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:34:58.960541 26327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:34:58.969074 26332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:58.969174 26335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:34:58.969300 26333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:34:58.969174 26327 server_base.cc:1034] running on GCE node
I20260620 14:34:58.969946 26327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:34:58.971012 26327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:34:58.972180 26327 hybrid_clock.cc:648] HybridClock initialized: now 1781966098972163 us; error 28 us; skew 500 ppm
Jun 20 14:34:58 dist-test-slave-gqzk krb5kdc[26314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:34:58.974858 26327 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:34:58.975741 26327 webserver.cc:460] Webserver started at http://127.4.58.126:44777/ using document root <none> and password file <none>
I20260620 14:34:58.976248 26327 fs_manager.cc:359] Metadata directory not provided
I20260620 14:34:58.976305 26327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:34:58.976466 26327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:34:58.978547 26327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "719c8aa2df534a2f995b1d1eeca344ff"
format_stamp: "Formatted at 2026-06-20 14:34:58 on dist-test-slave-gqzk"
server_key: "94ac2ea2409dfcce86fdb90f49414a56"
server_key_iv: "6fb5ea41e8eab862bec571b4f6a23dfb"
server_key_version: "encryptionkey@0"
I20260620 14:34:58.979063 26327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "719c8aa2df534a2f995b1d1eeca344ff"
format_stamp: "Formatted at 2026-06-20 14:34:58 on dist-test-slave-gqzk"
server_key: "94ac2ea2409dfcce86fdb90f49414a56"
server_key_iv: "6fb5ea41e8eab862bec571b4f6a23dfb"
server_key_version: "encryptionkey@0"
I20260620 14:34:58.982612 26327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:34:58.984946 26342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:34:58.986146 26327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:34:58.986258 26327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "719c8aa2df534a2f995b1d1eeca344ff"
format_stamp: "Formatted at 2026-06-20 14:34:58 on dist-test-slave-gqzk"
server_key: "94ac2ea2409dfcce86fdb90f49414a56"
server_key_iv: "6fb5ea41e8eab862bec571b4f6a23dfb"
server_key_version: "encryptionkey@0"
I20260620 14:34:58.986348 26327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:34:59.004935 26327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:34:59.008368 26327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:34:59.008545 26327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:34:59.015246 26327 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:42069
I20260620 14:34:59.015367 26394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:42069 every 8 connection(s)
I20260620 14:34:59.016209 26327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:34:59.019130 26395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:34:59.024943 26395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff: Bootstrap starting.
I20260620 14:34:59.026346 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26327
I20260620 14:34:59.026475 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:34:59.026775 4329 external_mini_cluster.cc:1428] Setting key be8604886ab7d6e4acd79325636b607c
I20260620 14:34:59.027679 26395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:34:59.028561 26395 log.cc:824] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:34:59.030928 26395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff: No bootstrap required, opened a new log
I20260620 14:34:59.034683 26395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } }
I20260620 14:34:59.034921 26395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:34:59.034974 26395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719c8aa2df534a2f995b1d1eeca344ff, State: Initialized, Role: FOLLOWER
I20260620 14:34:59.035461 26395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } }
I20260620 14:34:59.035594 26395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:34:59.035652 26395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:34:59.035749 26395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:34:59.036809 26395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } }
I20260620 14:34:59.037156 26395 leader_election.cc:302] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 719c8aa2df534a2f995b1d1eeca344ff; no voters:
I20260620 14:34:59.037346 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:59.027974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59377 (local address 127.4.58.126:42069)
0620 14:34:59.028458 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:59.028477 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:59.028521 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:34:59.029340 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:59.029360 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:59.029795 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:59.030195 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:59.030616 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:59.032062 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:59.033059 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:59.034322 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:59.034492 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:59.035342 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:59.035349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:59.035366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:59.035828 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:59.035886 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:59.035910 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:59.035993 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:59.036184 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:34:59.036476 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20260620 14:34:59.037475 26395 leader_election.cc:288] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:34:59.037652 26400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:34:59.037842 26400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [term 1 LEADER]: Becoming Leader. State: Replica: 719c8aa2df534a2f995b1d1eeca344ff, State: Running, Role: LEADER
I20260620 14:34:59.038218 26400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } }
I20260620 14:34:59.039420 26400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 719c8aa2df534a2f995b1d1eeca344ff. Latest consensus state: current_term: 1 leader_uuid: "719c8aa2df534a2f995b1d1eeca344ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } } }
I20260620 14:34:59.039547 26400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:59.039736 26395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:34:59.039943 26404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:34:59.040146 26402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "719c8aa2df534a2f995b1d1eeca344ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719c8aa2df534a2f995b1d1eeca344ff" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 42069 } } }
I20260620 14:34:59.040309 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:34:59.042292 26402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:34:59.043201 26404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:34:59.048939 26404 catalog_manager.cc:1347] Generated new cluster ID: d54deaccd91c4c20a0365687c18ce24a
I20260620 14:34:59.049062 26404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:34:59.053907 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:34:59.042488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33237 (local address 127.4.58.126:42069)
0620 14:34:59.042590 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:34:59.042594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:34:59.045381 (+ 2787us) server_negotiation.cc:407] Connection header received
0620 14:34:59.045518 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:34:59.045521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:34:59.045570 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:34:59.045647 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:34:59.045989 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:59.046745 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:59.047898 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:34:59.048225 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:34:59.048329 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:34:59.053293 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:34:59.053297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:34:59.053299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:34:59.053315 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:34:59.053352 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:34:59.053400 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:34:59.053569 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:34:59.053685 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:34:59.053754 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:34:59.061435 26404 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:34:59.062759 26404 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:34:59.069137 26404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 719c8aa2df534a2f995b1d1eeca344ff: Generated new TSK 0
I20260620 14:34:59.069988 26404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:34:59.075503 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26327
2026-06-20T14:34:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:34:59 dist-test-slave-gqzk krb5kdc[26426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:34:59 dist-test-slave-gqzk krb5kdc[26426](info): set up 2 sockets
Jun 20 14:34:59 dist-test-slave-gqzk krb5kdc[26426](info): commencing operation
krb5kdc: starting...
W20260620 14:35:00.025488 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:35:00 dist-test-slave-gqzk krb5kdc[26426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:35:00.151510 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34365
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:34365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:00.261075 26439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:00.261356 26439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:00.261399 26439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:00.265444 26439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:00.265518 26439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:00.265537 26439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:00.265553 26439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:00.265568 26439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:00.270972 26439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:34365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:34365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:00.272089 26439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:00.273160 26439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:00.281878 26447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:00.281854 26444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:00.281930 26439 server_base.cc:1034] running on GCE node
W20260620 14:35:00.281862 26445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:00.282591 26439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:00.283717 26439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:00.284857 26439 hybrid_clock.cc:648] HybridClock initialized: now 1781966100284833 us; error 34 us; skew 500 ppm
Jun 20 14:35:00 dist-test-slave-gqzk krb5kdc[26426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966100, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:00.287791 26439 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:35:00.288728 26439 webserver.cc:460] Webserver started at http://127.4.58.126:44871/ using document root <none> and password file <none>
I20260620 14:35:00.289331 26439 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:00.289399 26439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:00.289597 26439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:00.291738 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748"
format_stamp: "Formatted at 2026-06-20 14:35:00 on dist-test-slave-gqzk"
server_key: "237a403eb1215f7c40f2918fca7dc61c"
server_key_iv: "565e70ebd85b6c188660a8cc1dce5dbc"
server_key_version: "encryptionkey@0"
I20260620 14:35:00.292311 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748"
format_stamp: "Formatted at 2026-06-20 14:35:00 on dist-test-slave-gqzk"
server_key: "237a403eb1215f7c40f2918fca7dc61c"
server_key_iv: "565e70ebd85b6c188660a8cc1dce5dbc"
server_key_version: "encryptionkey@0"
I20260620 14:35:00.296350 26439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:35:00.299131 26454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:00.300580 26439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:35:00.300736 26439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748"
format_stamp: "Formatted at 2026-06-20 14:35:00 on dist-test-slave-gqzk"
server_key: "237a403eb1215f7c40f2918fca7dc61c"
server_key_iv: "565e70ebd85b6c188660a8cc1dce5dbc"
server_key_version: "encryptionkey@0"
I20260620 14:35:00.300911 26439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:00.305560 26439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:00.309032 26439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:00.309211 26439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:00.315894 26439 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:34365
I20260620 14:35:00.315917 26506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:34365 every 8 connection(s)
I20260620 14:35:00.316840 26439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:00.317710 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26439
I20260620 14:35:00.317828 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:35:00.318105 4329 external_mini_cluster.cc:1428] Setting key 09506a149b0b75566ad8bba5e057ec36
I20260620 14:35:00.319893 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:00.323333 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:00.319451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46351 (local address 127.4.58.126:34365)
0620 14:35:00.319857 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:00.319874 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:00.319909 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:35:00.320572 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:00.320588 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:00.320939 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:00.321228 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:00.322062 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:00.322066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:00.322078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:00.322088 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:00.322144 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:00.322162 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:00.322285 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:00.322425 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:35:00.322578 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260620 14:35:00.326150 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:00.326270 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748: Bootstrap starting.
I20260620 14:35:00.328946 26507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:00.329857 26507 log.cc:824] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:00.330001 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:00.328243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51117 (local address 127.4.58.126:34365)
0620 14:35:00.328402 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:00.328406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:00.328615 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:35:00.328825 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:00.328827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:00.328860 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:00.328932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:00.329626 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:00.329629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:00.329631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:35:00.329641 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:00.329668 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:35:00.329678 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:00.329787 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:00.329851 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:35:00.329884 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 14:35:00.331875 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748: No bootstrap required, opened a new log
I20260620 14:35:00.335462 26507 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } }
I20260620 14:35:00.335729 26507 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:35:00.335791 26507 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6210c9fc6f94dbfa7fbcdfc8624b748, State: Initialized, Role: FOLLOWER
I20260620 14:35:00.336256 26507 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } }
I20260620 14:35:00.336395 26507 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:35:00.336460 26507 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:35:00.336560 26507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:00.337553 26507 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } }
I20260620 14:35:00.337925 26507 leader_election.cc:302] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6210c9fc6f94dbfa7fbcdfc8624b748; no voters:
I20260620 14:35:00.338263 26507 leader_election.cc:288] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:35:00.338423 26517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:00.338642 26517 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [term 1 LEADER]: Becoming Leader. State: Replica: d6210c9fc6f94dbfa7fbcdfc8624b748, State: Running, Role: LEADER
I20260620 14:35:00.338953 26517 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } }
I20260620 14:35:00.339584 26507 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:00.340740 26518 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } } }
I20260620 14:35:00.340879 26518 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:00.341161 26519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6210c9fc6f94dbfa7fbcdfc8624b748. Latest consensus state: current_term: 1 leader_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6210c9fc6f94dbfa7fbcdfc8624b748" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 34365 } } }
I20260620 14:35:00.341295 26519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:00.341311 26526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:00.344156 26526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:00.349291 26526 catalog_manager.cc:1347] Generated new cluster ID: b062a29f0025425a90c4918cdf4bc8f0
I20260620 14:35:00.349365 26526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:00.359076 26526 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:00.359421 26526 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:00.364238 26526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6210c9fc6f94dbfa7fbcdfc8624b748: Generated new TSK 0
I20260620 14:35:00.365000 26526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:35:00.374374 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26439
2026-06-20T14:35:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1298 ms)
[----------] 18 tests from AuthTokenIssuingTest (23836 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:00 dist-test-slave-gqzk krb5kdc[26538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:00 dist-test-slave-gqzk krb5kdc[26538](info): set up 2 sockets
Jun 20 14:35:00 dist-test-slave-gqzk krb5kdc[26538](info): commencing operation
krb5kdc: starting...
W20260620 14:35:01.333107 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:35:01.456139 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45513
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:01.567096 26551 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:01.567416 26551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:01.567461 26551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:01.567526 26551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:01.571715 26551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:01.571792 26551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:01.571821 26551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:01.571841 26551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:01.571862 26551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:01.577385 26551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:01.578540 26551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:01.579559 26551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:01.588095 26556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:01.588081 26559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:01.588291 26551 server_base.cc:1034] running on GCE node
W20260620 14:35:01.588081 26557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:01.589097 26551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:01.590317 26551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:01.591542 26551 hybrid_clock.cc:648] HybridClock initialized: now 1781966101591516 us; error 33 us; skew 500 ppm
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:01.594602 26551 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:35:01.595575 26551 webserver.cc:460] Webserver started at http://127.4.58.126:40853/ using document root <none> and password file <none>
I20260620 14:35:01.596133 26551 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:01.596199 26551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:01.596388 26551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:01.598549 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "8bd54fe09aaa44699e9c65778b28ef42"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "ec78db9c9cdc732c8116a737c175e8a4"
server_key_iv: "0a7d550d7aa215b6879a8db6035a12a2"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.599112 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "8bd54fe09aaa44699e9c65778b28ef42"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "ec78db9c9cdc732c8116a737c175e8a4"
server_key_iv: "0a7d550d7aa215b6879a8db6035a12a2"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.603049 26551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 14:35:01.605522 26566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:01.606832 26551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:35:01.606957 26551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "8bd54fe09aaa44699e9c65778b28ef42"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "ec78db9c9cdc732c8116a737c175e8a4"
server_key_iv: "0a7d550d7aa215b6879a8db6035a12a2"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.607055 26551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:01.625591 26551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:01.628652 26551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:01.628857 26551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:01.635694 26551 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:45513
I20260620 14:35:01.635941 26618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:45513 every 8 connection(s)
I20260620 14:35:01.636674 26551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:01.639546 26619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:01.642725 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26551
I20260620 14:35:01.642844 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:35:01.643081 4329 external_mini_cluster.cc:1428] Setting key c652f1b6b6f65906ab3c8d1deb5fc28e
I20260620 14:35:01.643447 26619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.58.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.125@KRBTEST.COM" created.
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:01.658941 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.645832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45513 (local address 127.0.0.1:53706)
0620 14:35:01.646434 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.646474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:01.647308 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:01.647978 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:01.647984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:01.648429 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:01.648681 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.648694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.649851 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.649853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.650640 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.650650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.651261 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.651265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.651394 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.651986 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:01.652005 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:01.653949 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:01.655931 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.655941 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.655952 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.656241 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.656561 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.656564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.656566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.656725 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.657361 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:01.657369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:01.657544 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:35:01.657745 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:35:01.658121 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":67,"spinlock_wait_cycles":44032,"thread_start_us":141,"threads_started":1}
I20260620 14:35:01.658939 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.645831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.4.58.126:45513)
0620 14:35:01.646292 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:01.646311 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:01.646579 (+ 268us) server_negotiation.cc:407] Connection header received
0620 14:35:01.647609 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:01.647614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:01.647801 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:01.647966 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:01.648837 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.649713 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.650746 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.651111 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.651267 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.654118 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:01.654124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:01.654136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:01.654160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:01.655821 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.656357 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.656362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.656364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.656473 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.656846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.656850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.656852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.657247 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:01.657350 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:01.657793 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 14:35:01.658035 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":297,"spinlock_wait_cycles":1280,"thread_start_us":137,"threads_started":1}
I20260620 14:35:01.661505 26619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:01.662358 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.662032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40250)
0620 14:35:01.662190 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.662277 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:35:01.662578 26567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
I20260620 14:35:01.663702 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.663442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40256)
0620 14:35:01.663583 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.663639 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:35:01.663882 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:37123: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
I20260620 14:35:01.668190 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:37123
--webserver_interface=127.4.58.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:01.713099 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } attempt: 1
I20260620 14:35:01.714059 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.713721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40272)
0620 14:35:01.713887 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.713964 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:35:01.714798 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.714567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40282)
0620 14:35:01.714666 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.714720 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:35:01.714917 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:37123: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
W20260620 14:35:01.779184 26626 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:01.779601 26626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:01.779650 26626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:01.779707 26626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:35:01.783115 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } attempt: 2
I20260620 14:35:01.784029 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.783704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40298)
0620 14:35:01.783858 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.783924 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:35:01.784451 26626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:01.784525 26626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:01.784560 26626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:01.784590 26626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:01.784618 26626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:01.784897 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.784626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40302)
0620 14:35:01.784781 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.784835 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:35:01.785044 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:37123: Network error: Client connection negotiation failed: client connection to 127.4.58.125:37123: connect: Connection refused (error 111)
I20260620 14:35:01.790298 26626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.58.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:01.791320 26626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:01.792279 26626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:01.800662 26631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:01.800864 26626 server_base.cc:1034] running on GCE node
W20260620 14:35:01.800688 26634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:01.800675 26632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:01.801579 26626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:01.802695 26626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:01.803862 26626 hybrid_clock.cc:648] HybridClock initialized: now 1781966101803841 us; error 35 us; skew 500 ppm
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:01.806638 26626 init.cc:377] Logged in from keytab as kudu/127.4.58.125@KRBTEST.COM (short username kudu)
I20260620 14:35:01.807550 26626 webserver.cc:460] Webserver started at http://127.4.58.125:45707/ using document root <none> and password file <none>
I20260620 14:35:01.808082 26626 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:01.808147 26626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:01.808339 26626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:01.810503 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data/instance:
uuid: "f284af56ca604c63bfad042d45eb9d02"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "95ef6c9eabda70b0c38f5ce369766210"
server_key_iv: "5587294971d57809f1e186e463d9e27e"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.811069 26626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal/instance:
uuid: "f284af56ca604c63bfad042d45eb9d02"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "95ef6c9eabda70b0c38f5ce369766210"
server_key_iv: "5587294971d57809f1e186e463d9e27e"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.814877 26626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:35:01.817220 26641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:01.818486 26626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:35:01.818607 26626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "f284af56ca604c63bfad042d45eb9d02"
format_stamp: "Formatted at 2026-06-20 14:35:01 on dist-test-slave-gqzk"
server_key: "95ef6c9eabda70b0c38f5ce369766210"
server_key_iv: "5587294971d57809f1e186e463d9e27e"
server_key_version: "encryptionkey@0"
I20260620 14:35:01.818704 26626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:01.835318 26626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:01.838630 26626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:01.838824 26626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:01.845829 26626 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.125:37123
I20260620 14:35:01.846076 26693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.125:37123 every 8 connection(s)
I20260620 14:35:01.846834 26626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
I20260620 14:35:01.849640 26694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:01.853474 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:01.854696 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26626
I20260620 14:35:01.854813 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-1/wal/instance
I20260620 14:35:01.855105 4329 external_mini_cluster.cc:1428] Setting key bfc546b481f05a9ae9a576c9435c483a
WARNING: no policy specified for kudu/127.4.58.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.124@KRBTEST.COM" created.
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:01.870500 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.855418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.4.58.126:45513)
0620 14:35:01.855545 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:01.855549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:01.856641 (+ 1092us) server_negotiation.cc:407] Connection header received
0620 14:35:01.857908 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:01.857911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:01.857961 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:01.858064 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:01.859351 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.860173 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.861454 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.861821 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.861934 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.866768 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:01.866772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:01.866774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:01.866807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:01.868217 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.868832 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.868835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.868836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.868930 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.869280 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.869282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.869283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.869505 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:01.869596 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:01.870264 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 14:35:01.870373 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:01.871714 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.855928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45513 (local address 127.0.0.1:53718)
0620 14:35:01.856480 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.856525 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:01.857586 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:01.858254 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:01.858262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:01.858604 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:01.859110 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.859128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.860317 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.860319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.861283 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.861293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.861993 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.861997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.862142 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.862731 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:01.862747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:01.866572 (+ 3825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:01.868341 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.868349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.868366 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.868711 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.869006 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.869009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.869011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.869192 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.869622 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:01.869630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:01.869993 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 14:35:01.870218 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 14:35:01.870588 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260620 14:35:01.873198 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:01.876209 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } attempt: 3
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:01 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
I20260620 14:35:01.884305 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:44057
--webserver_interface=127.4.58.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:01.884505 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.873648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40304)
0620 14:35:01.873833 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.873845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:01.873912 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:01.874448 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:01.874450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:01.874675 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:01.874795 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.874800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.875838 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.875841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.876569 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.876577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.877819 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.877822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.877914 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.878487 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:01.878502 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:01.879766 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:01.882545 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.882548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.882550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.882807 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.883198 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.883201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.883204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.883295 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.883938 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:01.883941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:01.884040 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:35:01.884147 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:35:01.884292 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 14:35:01.885100 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.873759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40304 (local address 127.4.58.125:37123)
0620 14:35:01.874047 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:01.874056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:01.874074 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:35:01.874108 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:01.874112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:01.874267 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:01.874385 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:01.874924 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.875732 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.876681 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.877280 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.877496 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.879966 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:01.879980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:01.879999 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:01.880041 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:01.882439 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.882972 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.882978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.882980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.883075 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.883444 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.883447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.883449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.883791 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:01.883970 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:01.884795 (+ 825us) server_negotiation.cc:299] Negotiation successful
0620 14:35:01.884941 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
I20260620 14:35:01.885824 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.876699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40310)
0620 14:35:01.876889 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.876905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:01.876988 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:01.877394 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:01.877397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:01.877538 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:01.877632 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.877637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.878494 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.878497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.879762 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:01.879773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.880450 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.880453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:01.880595 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.880991 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:01.881005 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:01.882418 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:01.884155 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.884158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.884160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.884354 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.884641 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:01.884644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:01.884646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:01.884822 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:01.885370 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:01.885372 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:01.885456 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:35:01.885537 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:35:01.885668 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 14:35:01.885885 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.876803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.4.58.125:37123)
0620 14:35:01.877108 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:01.877116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:01.877128 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:35:01.877157 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:01.877160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:01.877278 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:01.877372 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:01.877753 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.878356 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.879927 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:01.880332 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:01.880418 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:01.882556 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:01.882559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:01.882562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:01.882586 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:01.884063 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.884461 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.884465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.884467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.884547 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:01.884929 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:01.884932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:01.884933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:01.885098 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:01.885187 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:01.885619 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 14:35:01.885713 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":239,"thread_start_us":129,"threads_started":1}
I20260620 14:35:01.887516 26619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:01.887607 26694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:01.888226 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.888000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50604)
0620 14:35:01.888105 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.888175 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:35:01.888491 26644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
I20260620 14:35:01.889382 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.889162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50616)
0620 14:35:01.889254 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.889336 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:35:01.889468 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.888269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50610)
0620 14:35:01.889338 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.889392 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":989}
W20260620 14:35:01.889786 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
I20260620 14:35:01.890206 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.889942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50628)
0620 14:35:01.890087 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.890139 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 14:35:01.890352 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
I20260620 14:35:01.921514 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 1
I20260620 14:35:01.922421 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.922062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50638)
0620 14:35:01.922223 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.922329 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:35:01.923146 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.922943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50640)
0620 14:35:01.923044 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.923087 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:35:01.923282 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
I20260620 14:35:01.939090 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 1
I20260620 14:35:01.939997 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.939642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50650)
0620 14:35:01.939850 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.939917 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 14:35:01.940896 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.940618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50662)
0620 14:35:01.940784 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.940835 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 14:35:01.941004 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
W20260620 14:35:01.997431 26701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:01.997721 26701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:01.997774 26701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:01.997839 26701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:35:01.998430 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 2
I20260620 14:35:01.999307 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.998945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50678)
0620 14:35:01.999133 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.999198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:35:01.999991 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:01.999786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50682)
0620 14:35:01.999890 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:01.999934 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:35:02.000104 26619 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
W20260620 14:35:02.002241 26701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:02.002298 26701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:02.002326 26701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:02.002357 26701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:02.002385 26701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:02.007781 26701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45513,127.4.58.125:37123,127.4.58.124:44057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:44057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.58.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:02.008956 26701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:02.009169 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 2
I20260620 14:35:02.009939 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.009620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50688)
0620 14:35:02.009764 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.009831 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:35:02.010048 26701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:02.010759 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.010496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50690)
0620 14:35:02.010630 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.010699 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:35:02.010929 26694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:44057: Network error: Client connection negotiation failed: client connection to 127.4.58.124:44057: connect: Connection refused (error 111)
W20260620 14:35:02.018802 26708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:02.018810 26707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:02.018867 26701 server_base.cc:1034] running on GCE node
W20260620 14:35:02.018779 26710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:02.019649 26701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:02.020890 26701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:02.022051 26701 hybrid_clock.cc:648] HybridClock initialized: now 1781966102022019 us; error 48 us; skew 500 ppm
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:02.024966 26701 init.cc:377] Logged in from keytab as kudu/127.4.58.124@KRBTEST.COM (short username kudu)
I20260620 14:35:02.025913 26701 webserver.cc:460] Webserver started at http://127.4.58.124:44223/ using document root <none> and password file <none>
I20260620 14:35:02.026482 26701 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:02.026547 26701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:02.026758 26701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:02.029145 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data/instance:
uuid: "2fbdca560fdd42709c95995510f88c40"
format_stamp: "Formatted at 2026-06-20 14:35:02 on dist-test-slave-gqzk"
server_key: "e9246507274a82bcc5589a54c75b6f6f"
server_key_iv: "a54af44f541e39894285be77141b88aa"
server_key_version: "encryptionkey@0"
I20260620 14:35:02.029731 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal/instance:
uuid: "2fbdca560fdd42709c95995510f88c40"
format_stamp: "Formatted at 2026-06-20 14:35:02 on dist-test-slave-gqzk"
server_key: "e9246507274a82bcc5589a54c75b6f6f"
server_key_iv: "a54af44f541e39894285be77141b88aa"
server_key_version: "encryptionkey@0"
I20260620 14:35:02.033595 26701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:35:02.036110 26717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:02.037436 26701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:35:02.037546 26701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "2fbdca560fdd42709c95995510f88c40"
format_stamp: "Formatted at 2026-06-20 14:35:02 on dist-test-slave-gqzk"
server_key: "e9246507274a82bcc5589a54c75b6f6f"
server_key_iv: "a54af44f541e39894285be77141b88aa"
server_key_version: "encryptionkey@0"
I20260620 14:35:02.037662 26701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:02.055222 26701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:02.058584 26701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:02.058758 26701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:02.065965 26701 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.124:44057
I20260620 14:35:02.066229 26769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.124:44057 every 8 connection(s)
I20260620 14:35:02.066968 26701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
I20260620 14:35:02.069868 26770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:02.072038 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26701
I20260620 14:35:02.072145 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965824702348-4329-0/minicluster-data/master-2/wal/instance
I20260620 14:35:02.072472 4329 external_mini_cluster.cc:1428] Setting key c30e4f2d0d60a896ef72b07eed714545
I20260620 14:35:02.073982 26770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:35:02.085929 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.075813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.4.58.126:45513)
0620 14:35:02.075946 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.075950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.076385 (+ 435us) server_negotiation.cc:407] Connection header received
0620 14:35:02.076647 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.076650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.076707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.076889 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.077658 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.078159 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.079134 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.079358 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.079446 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.082076 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.082081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.082086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.082115 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.083708 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.084202 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.084205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.084208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.084286 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.084617 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.084620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.084622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.084942 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.085063 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.085608 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.085741 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:35:02.086887 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.075729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45513 (local address 127.0.0.1:53730)
0620 14:35:02.076231 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.076291 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:02.076494 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:02.076958 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:02.076964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:02.077342 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:02.077507 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.077517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.078276 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.078279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.079014 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.079023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.079503 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.079507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.079608 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.080158 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:02.080174 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:02.081891 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:02.083824 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.083834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.083846 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.084100 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.084372 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.084375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.084385 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.084528 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.085065 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:02.085073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:02.085371 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 14:35:02.085545 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:35:02.085905 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":7,"thread_start_us":125,"threads_started":1}
I20260620 14:35:02.086889 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.073970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.4.58.124:44057)
0620 14:35:02.074431 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.074449 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.074489 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:35:02.075210 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.075228 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.075680 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.076083 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.077109 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.078127 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.079039 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.079425 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.079573 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.082420 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.082426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.082440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.082468 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.083983 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.084420 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.084425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.084427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.084538 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.084847 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.084850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.084852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.085404 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.085606 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.085866 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.086035 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":6272,"thread_start_us":124,"threads_started":1}
I20260620 14:35:02.088675 26770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:02.096987 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.089354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.4.58.125:37123)
0620 14:35:02.089484 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.089488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.089504 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:35:02.089570 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.089573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.089619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.089681 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.090158 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.090760 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.091611 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.091870 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.091953 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.094114 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.094117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.094121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.094155 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.095476 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.095855 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.095859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.095860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.095960 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.096201 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.096204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.096206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.096377 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.096461 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.096701 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.096838 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:35:02.096987 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.089210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:37123 (local address 127.0.0.1:40316)
0620 14:35:02.089352 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.089366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:02.089484 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:02.089718 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:02.089721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:02.089900 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:02.090024 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.090031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.090874 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.090877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.091498 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.091508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.091970 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.091973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.092071 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.092593 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:02.092607 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:02.093944 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:02.095579 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.095582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.095584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.095771 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.096045 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.096048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.096050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.096125 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.096465 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:02.096468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:02.096558 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 14:35:02.096641 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:35:02.096810 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:35:02.097813 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.089517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.4.58.126:45513)
0620 14:35:02.089660 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.089664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.089829 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:35:02.089939 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.089941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.089977 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.090036 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.090618 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.091202 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.092076 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.092306 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.092401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.094759 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.094772 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.094774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.094798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.096086 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.096470 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.096473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.096475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.096548 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.096827 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.096830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.096831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.097016 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.097243 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.097566 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.097673 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:35:02.097853 26770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:02.099246 26619 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 3
I20260620 14:35:02.102109 26694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } attempt: 3
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966102, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
Jun 20 14:35:02 dist-test-slave-gqzk krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:35:02.108124 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.098334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50706)
0620 14:35:02.098442 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.098456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:02.098524 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:02.099158 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:02.099162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:02.099346 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:02.099457 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.099463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.100326 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.100329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.101054 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.101064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.101625 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.101629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.101706 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.102226 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:02.102240 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:02.104230 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:02.106328 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.106331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.106334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.106657 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.107094 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.107097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.107099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.107174 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.107575 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:02.107578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:02.107657 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:35:02.107821 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:35:02.107957 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260620 14:35:02.108896 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.098467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.4.58.124:44057)
0620 14:35:02.098564 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.098569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.098580 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:02.098612 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.098615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.098653 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.098729 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.099558 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.100232 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.101189 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.101532 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.101614 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.104347 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.104351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.104353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.104374 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.106218 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.106897 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.106900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.106902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.107004 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.107261 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.107264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.107266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.107453 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.107542 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.108365 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.108467 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":1280}
I20260620 14:35:02.109293 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.098930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.4.58.125:37123)
0620 14:35:02.099087 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.099090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.099102 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:35:02.099195 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.099197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.099236 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.099317 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.099990 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.100612 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.101470 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.101795 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.101927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.104444 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.104448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.104451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.104474 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.107182 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.107639 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.107642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.107644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.107722 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.108024 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.108028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.108029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.108230 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.108961 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.109041 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.109155 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:35:02.110390 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.099629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50716)
0620 14:35:02.099760 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.099773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:02.099840 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:02.100373 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:02.100376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:02.100537 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:02.100631 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.100637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.102032 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.102035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.102735 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.102746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.103906 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.103910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.103996 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.104415 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:02.104428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:02.105723 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:02.107920 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.107923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.107926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.108158 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.109294 (+ 1136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.109297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.109299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.109394 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.109908 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:02.109911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:02.110010 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:35:02.110087 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:35:02.110217 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:35:02.111456 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.099850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.4.58.124:44057)
0620 14:35:02.100146 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.100154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.100166 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:35:02.100195 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.100197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.100231 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.100348 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.100791 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.101911 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.103403 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.103773 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.104195 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.105847 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.105851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.105853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.105888 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.107797 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.108587 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.108591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.108592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.108666 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.109558 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.109561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.109563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.109762 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.111110 (+ 1348us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.111181 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.111302 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":127,"threads_started":1}
I20260620 14:35:02.112171 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.103191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.4.58.124:44057)
0620 14:35:02.103516 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.103522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.103532 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:02.103558 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.103560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.103592 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.103668 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.104070 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.105110 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.106144 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.106460 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.106857 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.108449 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.108452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.108455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.108476 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.110567 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.110997 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.111000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.111001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.111078 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.111366 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.111369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.111371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.111610 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.111700 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.111972 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.112057 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"spinlock_wait_cycles":896,"thread_start_us":134,"threads_started":1}
I20260620 14:35:02.112190 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.102467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:44057 (local address 127.0.0.1:50728)
0620 14:35:02.102973 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:02.102986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:02.103053 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:02.103704 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:02.103708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:02.103858 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:02.103944 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.103951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.105231 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.105235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.106006 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:02.106018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.106586 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.106590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:02.106674 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.107001 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:02.107014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:02.108319 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:02.110670 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.110674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.110676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.110900 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.111204 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:02.111207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:02.111209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:02.111287 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:02.111711 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:02.111714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:02.111812 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:35:02.111918 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:35:02.112047 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435}
I20260620 14:35:02.112573 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: Bootstrap starting.
I20260620 14:35:02.115024 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:02.115526 26770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:02.115589 26694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: Bootstrap starting.
I20260620 14:35:02.115589 26619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: Bootstrap starting.
I20260620 14:35:02.116441 26770 log.cc:824] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:02.116369 26593 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8bd54fe09aaa44699e9c65778b28ef42"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53736
I20260620 14:35:02.118297 26694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:02.118296 26619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:02.119148 26619 log.cc:824] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:02.119148 26694 log.cc:824] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:02.119654 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: No bootstrap required, opened a new log
I20260620 14:35:02.121230 26619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: No bootstrap required, opened a new log
I20260620 14:35:02.123569 26619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.123775 26619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd54fe09aaa44699e9c65778b28ef42, State: Initialized, Role: FOLLOWER
I20260620 14:35:02.124074 26619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.123972 26770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.124202 26770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbdca560fdd42709c95995510f88c40, State: Initialized, Role: FOLLOWER
I20260620 14:35:02.124514 26770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.124522 26780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.124641 26780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.124986 26781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.125125 26781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.125487 26619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:02.125589 26593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:35:02.125702 26593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:02.125808 26770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:02.127255 26593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.129793 26593 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [CANDIDATE]: Term 1 election: Requested vote from peers f284af56ca604c63bfad042d45eb9d02 (127.4.58.125:37123), 2fbdca560fdd42709c95995510f88c40 (127.4.58.124:44057)
I20260620 14:35:02.130134 26694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: No bootstrap required, opened a new log
W20260620 14:35:02.130363 26572 tablet.cc:2369] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:35:02.132812 26803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:02.132973 26803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:02.131319 26551 master_runner.cc:186] Error getting master registration for 127.4.58.125:37123: OK, instance_id { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" instance_seqno: 1781966101843841 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:02.134372 26694 raft_consensus.cc:348] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.134606 26551 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:02.134632 26694 raft_consensus.cc:729] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f284af56ca604c63bfad042d45eb9d02, State: Initialized, Role: FOLLOWER
I20260620 14:35:02.134994 26694 consensus_queue.cc:260] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.136416 26694 sys_catalog.cc:564] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:02.136936 26667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bd54fe09aaa44699e9c65778b28ef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f284af56ca604c63bfad042d45eb9d02"
I20260620 14:35:02.137177 26667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:02.138434 26742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bd54fe09aaa44699e9c65778b28ef42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2fbdca560fdd42709c95995510f88c40"
I20260620 14:35:02.138628 26667 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bd54fe09aaa44699e9c65778b28ef42 in term 1.
I20260620 14:35:02.138720 26742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:02.139186 26567 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bd54fe09aaa44699e9c65778b28ef42, f284af56ca604c63bfad042d45eb9d02; no voters:
I20260620 14:35:02.139488 26780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:02.139705 26780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd54fe09aaa44699e9c65778b28ef42, State: Running, Role: LEADER
I20260620 14:35:02.140039 26742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bd54fe09aaa44699e9c65778b28ef42 in term 1.
I20260620 14:35:02.140049 26780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } }
I20260620 14:35:02.142011 26780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd54fe09aaa44699e9c65778b28ef42. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.142146 26780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:02.139756 26701 master_runner.cc:186] Error getting master registration for 127.4.58.125:37123: OK, instance_id { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" instance_seqno: 1781966101843841 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:02.142925 26701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:35:02.144796 26802 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:02.144862 26802 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:02.145994 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.133226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.4.58.126:45513)
0620 14:35:02.133346 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.133351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.134471 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 14:35:02.134612 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.134615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.134665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.134717 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.135769 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.136488 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.140215 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.140516 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.140594 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.142497 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.142501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.142503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.142529 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.144074 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.144529 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.144534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.144535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.144619 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.144955 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.144958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.144960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.145196 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.145314 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.145731 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.145848 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:35:02.147593 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.133852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.4.58.125:37123)
0620 14:35:02.133972 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.133976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.135951 (+ 1975us) server_negotiation.cc:407] Connection header received
0620 14:35:02.136117 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.136120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.136215 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.136268 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.137314 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.137908 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.141289 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.141644 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.141717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.144005 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.144009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.144011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.144035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.145652 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.146310 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.146314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.146316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.146399 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.146684 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.146687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.146688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.146895 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.147012 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.147315 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.147448 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:35:02.152575 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:02.138474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.4.58.124:44057)
0620 14:35:02.138600 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.138604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.138618 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:35:02.138656 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.138659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.138704 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.138783 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.145784 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.146549 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.147901 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:02.148178 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:02.148278 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:02.149744 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:02.149748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:02.149751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:02.149775 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:02.151045 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.151432 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.151436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.151438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.151513 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:02.151750 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:02.151754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:02.151755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:02.151935 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:02.152068 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:02.152308 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:35:02.152425 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:35:02.153100 26825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:02.153972 26825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:02.157094 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.157246 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.159339 26667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [term 1 FOLLOWER]: Refusing update from remote peer 8bd54fe09aaa44699e9c65778b28ef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:02.159350 26742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [term 1 FOLLOWER]: Refusing update from remote peer 8bd54fe09aaa44699e9c65778b28ef42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:02.160037 26823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:02.161952 26823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:02.166088 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd54fe09aaa44699e9c65778b28ef42. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.166221 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.170435 26781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd54fe09aaa44699e9c65778b28ef42. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.170576 26781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:35:02.173152 26827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:02.173312 26827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:02.175261 26780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.175422 26780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:02.175529 26828 mvcc.cc:205] Tried to move back new op lower bound from 7298933154437337088 to 7298933154369560576. Current Snapshot: MvccSnapshot[applied={T|T < 7298933154437337088}]
I20260620 14:35:02.175644 26781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.175288 26823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.175756 26781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.175774 26823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42 [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:02.176071 26832 mvcc.cc:205] Tried to move back new op lower bound from 7298933154437337088 to 7298933154369560576. Current Snapshot: MvccSnapshot[applied={T|T < 7298933154437337088}]
I20260620 14:35:02.176692 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bd54fe09aaa44699e9c65778b28ef42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bd54fe09aaa44699e9c65778b28ef42" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45513 } } peers { permanent_uuid: "f284af56ca604c63bfad042d45eb9d02" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 37123 } } peers { permanent_uuid: "2fbdca560fdd42709c95995510f88c40" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 44057 } } }
I20260620 14:35:02.176838 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:02.177211 26830 mvcc.cc:205] Tried to move back new op lower bound from 7298933154437337088 to 7298933154369560576. Current Snapshot: MvccSnapshot[applied={T|T < 7298933154437337088}]
I20260620 14:35:02.179780 26825 catalog_manager.cc:1347] Generated new cluster ID: 2aa736e5178849d2b4a037a36c23ed24
I20260620 14:35:02.179849 26825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:02.201924 26825 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:02.203292 26825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:02.226154 26825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd54fe09aaa44699e9c65778b28ef42: Generated new TSK 0
I20260620 14:35:02.227653 26825 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:35:02.265918 26776 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.262554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.4.58.124:44057)
0620 14:35:02.262693 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.262698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.264815 (+ 2117us) server_negotiation.cc:407] Connection header received
0620 14:35:02.265028 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.265032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.265085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.265135 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.265778 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50756: BlockingRecv error: recv got EOF from 127.0.0.1:50756 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:35:02.267359 26622 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.263424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.4.58.126:45513)
0620 14:35:02.263539 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.263543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.266425 (+ 2882us) server_negotiation.cc:407] Connection header received
0620 14:35:02.266597 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.266601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.266657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.266720 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.267227 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53754: BlockingRecv error: recv got EOF from 127.0.0.1:53754 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:35:02.268898 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.264159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.4.58.125:37123)
0620 14:35:02.264274 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.264279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.267852 (+ 3573us) server_negotiation.cc:407] Connection header received
0620 14:35:02.268105 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.268108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.268160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.268213 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.268726 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40348: BlockingRecv error: recv got EOF from 127.0.0.1:40348 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:35:02.273785 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.272126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.4.58.125:37123)
0620 14:35:02.272237 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.272242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.272702 (+ 460us) server_negotiation.cc:407] Connection header received
0620 14:35:02.272951 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.272955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.273035 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.273092 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.273650 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:35:02.279430 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.277751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.4.58.125:37123)
0620 14:35:02.277875 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.277879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.278436 (+ 557us) server_negotiation.cc:407] Connection header received
0620 14:35:02.278634 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.278637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.278709 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.278761 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.279307 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40366: BlockingRecv error: recv got EOF from 127.0.0.1:40366 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:35:02.285554 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.283836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.4.58.125:37123)
0620 14:35:02.283947 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.283952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.284477 (+ 525us) server_negotiation.cc:407] Connection header received
0620 14:35:02.284665 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.284668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.284731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.284881 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.285430 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40372: BlockingRecv error: recv got EOF from 127.0.0.1:40372 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:35:02.294167 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.292187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.4.58.125:37123)
0620 14:35:02.292362 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.292367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.293002 (+ 635us) server_negotiation.cc:407] Connection header received
0620 14:35:02.293246 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.293250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.293312 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.293369 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.294043 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40378: BlockingRecv error: recv got EOF from 127.0.0.1:40378 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 14:35:02.307029 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.305135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.4.58.125:37123)
0620 14:35:02.305274 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.305279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.305896 (+ 617us) server_negotiation.cc:407] Connection header received
0620 14:35:02.306133 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.306138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.306199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.306258 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.306895 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40382: BlockingRecv error: recv got EOF from 127.0.0.1:40382 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 14:35:02.328013 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.325805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.4.58.125:37123)
0620 14:35:02.325992 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.325996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.326750 (+ 754us) server_negotiation.cc:407] Connection header received
0620 14:35:02.326980 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.326984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.327035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.327107 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.327875 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40388: BlockingRecv error: recv got EOF from 127.0.0.1:40388 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 14:35:02.364346 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.363072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.4.58.125:37123)
0620 14:35:02.363223 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.363229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.363405 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:35:02.363559 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.363562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.363610 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.363680 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.364166 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:35:02.431941 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.430652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.4.58.125:37123)
0620 14:35:02.430831 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.430837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.431005 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:35:02.431161 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.431164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.431263 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.431339 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.431817 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40402: BlockingRecv error: recv got EOF from 127.0.0.1:40402 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:35:02.563428 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.562124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.4.58.125:37123)
0620 14:35:02.562260 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.562265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.562503 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:35:02.562670 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.562672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.562755 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.562831 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.563316 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40416: BlockingRecv error: recv got EOF from 127.0.0.1:40416 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:35:02.823181 26700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:02.821917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.4.58.125:37123)
0620 14:35:02.822036 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:02.822041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:02.822125 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:35:02.822383 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:02.822387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:02.822439 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:02.822518 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:02.823055 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40430: BlockingRecv error: recv got EOF from 127.0.0.1:40430 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:35:03.146175 26802 catalog_manager.cc:1259] Loaded cluster ID: 2aa736e5178849d2b4a037a36c23ed24
I20260620 14:35:03.146261 26802 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: loading cluster ID for follower catalog manager: success
I20260620 14:35:03.147957 26802 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: acquiring CA information for follower catalog manager: success
I20260620 14:35:03.149196 26802 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fbdca560fdd42709c95995510f88c40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:03.174693 26827 catalog_manager.cc:1259] Loaded cluster ID: 2aa736e5178849d2b4a037a36c23ed24
I20260620 14:35:03.174786 26827 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: loading cluster ID for follower catalog manager: success
I20260620 14:35:03.176507 26827 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: acquiring CA information for follower catalog manager: success
I20260620 14:35:03.177728 26827 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f284af56ca604c63bfad042d45eb9d02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:03.341686 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.337574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.4.58.125:37123)
0620 14:35:03.337852 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:03.337857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:03.337873 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:35:03.338058 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:03.338062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:03.338189 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:03.338287 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:03.338572 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.339172 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.340101 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.340333 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.340425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:03.340805 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:03.341270 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:03.341390 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:03.341460 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:35:03.341524 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":65,"threads_started":1}
I20260620 14:35:03.348553 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.344714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.4.58.124:44057)
0620 14:35:03.344962 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:03.344966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:03.344980 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:35:03.345085 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:03.345087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:03.345207 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:03.345306 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:03.345534 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.346069 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.346948 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.347269 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.347364 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:03.347707 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:03.348153 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:03.348247 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:03.348378 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:35:03.348431 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260620 14:35:03.355159 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.351505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.4.58.125:37123)
0620 14:35:03.351614 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:03.351618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:03.351943 (+ 325us) server_negotiation.cc:407] Connection header received
0620 14:35:03.352048 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:03.352051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:03.352105 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:03.352168 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:03.352371 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.353197 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.354196 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.354434 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.354522 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:03.354740 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:03.354808 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:03.354873 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:03.354967 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:35:03.355018 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:35:03.355814 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.351455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.4.58.124:44057)
0620 14:35:03.351563 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:03.351568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:03.351891 (+ 323us) server_negotiation.cc:407] Connection header received
0620 14:35:03.352043 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:03.352046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:03.352096 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:03.352139 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:03.352472 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.353248 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.354573 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.354900 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.354970 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:03.355382 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:03.355454 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:03.355497 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:03.355643 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:35:03.355695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:03.356359 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:03.351396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.4.58.126:45513)
0620 14:35:03.351706 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:03.351710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:03.351923 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:35:03.352143 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:03.352147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:03.352312 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:03.352477 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:03.352778 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.353345 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.354573 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:03.354920 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:03.355001 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:03.355432 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:03.355794 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:03.355877 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:03.356162 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:35:03.356223 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":58,"threads_started":1}
I20260620 14:35:03.365156 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26551
I20260620 14:35:03.370751 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26626
I20260620 14:35:03.375814 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26701
2026-06-20T14:35:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3002 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:35:03 dist-test-slave-gqzk krb5kdc[26921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:35:03 dist-test-slave-gqzk krb5kdc[26921](info): set up 2 sockets
Jun 20 14:35:03 dist-test-slave-gqzk krb5kdc[26921](info): commencing operation
krb5kdc: starting...
W20260620 14:35:04.324966 4329 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/test-user.keytab.
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.58.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
Entry for principal kudu/127.4.58.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab.
I20260620 14:35:04.451412 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45031
--webserver_interface=127.4.58.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:35:04.560925 26934 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:04.561234 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:04.561257 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:04.561295 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:04.565405 26934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:04.565486 26934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:04.565505 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:04.565521 26934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:04.565534 26934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:04.571390 26934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.126.keytab
--principal=kudu/127.4.58.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.126:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.58.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:04.572551 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:04.573616 26934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:04.582522 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:04.582542 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:04.582540 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:04.582597 26934 server_base.cc:1034] running on GCE node
I20260620 14:35:04.583408 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:04.584551 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:04.585740 26934 hybrid_clock.cc:648] HybridClock initialized: now 1781966104585718 us; error 64 us; skew 500 ppm
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:04.588718 26934 init.cc:377] Logged in from keytab as kudu/127.4.58.126@KRBTEST.COM (short username kudu)
I20260620 14:35:04.589838 26934 webserver.cc:460] Webserver started at http://127.4.58.126:34593/ using document root <none> and password file <none>
I20260620 14:35:04.590430 26934 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:04.590505 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:04.590716 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:04.592959 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data/instance:
uuid: "992adbcd6adf4d2981356b5e7c21ef6d"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "48f7cec4d0fe9f8209d2a76483eb2943"
server_key_iv: "b010ed7190b2fd2830cd2c24b147850f"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.593544 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance:
uuid: "992adbcd6adf4d2981356b5e7c21ef6d"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "48f7cec4d0fe9f8209d2a76483eb2943"
server_key_iv: "b010ed7190b2fd2830cd2c24b147850f"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.597477 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:35:04.600009 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:04.601459 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:35:04.601598 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal
uuid: "992adbcd6adf4d2981356b5e7c21ef6d"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "48f7cec4d0fe9f8209d2a76483eb2943"
server_key_iv: "b010ed7190b2fd2830cd2c24b147850f"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.601711 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:04.623126 26934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:04.626508 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:04.626726 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:04.634238 26934 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.126:45031
I20260620 14:35:04.634487 27001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.126:45031 every 8 connection(s)
I20260620 14:35:04.635251 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/data/info.pb
I20260620 14:35:04.637990 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 26934
I20260620 14:35:04.638103 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-0/wal/instance
I20260620 14:35:04.638121 27002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:04.638447 4329 external_mini_cluster.cc:1428] Setting key 62dde4eefad4b5a823f88d4ea9c10369
I20260620 14:35:04.642055 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.58.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.125@KRBTEST.COM" created.
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:04.657770 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.644460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45031 (local address 127.0.0.1:38038)
0620 14:35:04.645102 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.645137 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:04.645984 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:04.646651 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:04.646656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:04.647079 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:04.647421 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.647442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.648599 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.648601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.649362 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.649372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.649969 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.649972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.650126 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.650654 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:04.650671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:04.652415 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:04.654419 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.654430 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.654441 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.654751 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.655120 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.655123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.655125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.655273 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.655892 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:04.655898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:04.656035 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 14:35:04.656220 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:35:04.656616 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":96,"spinlock_wait_cycles":28416,"thread_start_us":161,"threads_started":1}
I20260620 14:35:04.657770 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.644459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.4.58.126:45031)
0620 14:35:04.644911 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:04.644927 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:04.645247 (+ 320us) server_negotiation.cc:407] Connection header received
0620 14:35:04.646274 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:04.646278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:04.646481 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:04.646640 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:04.647568 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.648488 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.649468 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.649828 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.649981 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.652605 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:04.652611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:04.652622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:04.652645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:04.654298 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.654887 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.654892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.654894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.655018 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.655358 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.655360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.655361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.655747 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:04.655851 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:04.656296 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 14:35:04.656554 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":305,"thread_start_us":117,"threads_started":1}
I20260620 14:35:04.660395 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:04.661204 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.660945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:54966)
0620 14:35:04.661067 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.661139 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:35:04.661389 26951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
I20260620 14:35:04.662243 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.661994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:54974)
0620 14:35:04.662124 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.662173 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:35:04.662391 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:35557: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
Entry for principal kudu/127.4.58.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab.
I20260620 14:35:04.664521 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:35557
--webserver_interface=127.4.58.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:04.711660 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } attempt: 1
I20260620 14:35:04.712616 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.712274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:54990)
0620 14:35:04.712464 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.712532 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 14:35:04.713438 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.713160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:54996)
0620 14:35:04.713322 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.713371 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:35:04.713562 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:35557: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
W20260620 14:35:04.773847 27009 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:04.774171 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:04.774206 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:04.774271 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:35:04.778373 27009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:04.778450 27009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:04.778475 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:04.778499 27009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:04.778518 27009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:04.781715 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } attempt: 2
I20260620 14:35:04.782608 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.782269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:55004)
0620 14:35:04.782445 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.782520 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 14:35:04.783464 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.783162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:55008)
0620 14:35:04.783346 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.783396 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 14:35:04.783591 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.125:35557: Network error: Client connection negotiation failed: client connection to 127.4.58.125:35557: connect: Connection refused (error 111)
I20260620 14:35:04.784317 27009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.125.keytab
--principal=kudu/127.4.58.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.125:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.58.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:04.785507 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:04.786517 27009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:35:04.795035 27017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:04.795040 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:04.795171 27009 server_base.cc:1034] running on GCE node
W20260620 14:35:04.795037 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:04.795882 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:04.797022 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:04.798214 27009 hybrid_clock.cc:648] HybridClock initialized: now 1781966104798184 us; error 31 us; skew 500 ppm
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:04.801070 27009 init.cc:377] Logged in from keytab as kudu/127.4.58.125@KRBTEST.COM (short username kudu)
I20260620 14:35:04.802011 27009 webserver.cc:460] Webserver started at http://127.4.58.125:34231/ using document root <none> and password file <none>
I20260620 14:35:04.802534 27009 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:04.802594 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:04.802867 27009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:04.805111 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data/instance:
uuid: "ac74ef54542c4900954dabe583d6131c"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "505093c55b27ccdb9c89ebe11bd4c2be"
server_key_iv: "2dfc4857a87381b4857a9ad58f3c5053"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.805641 27009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal/instance:
uuid: "ac74ef54542c4900954dabe583d6131c"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "505093c55b27ccdb9c89ebe11bd4c2be"
server_key_iv: "2dfc4857a87381b4857a9ad58f3c5053"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.809607 27009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:35:04.812155 27024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:04.813558 27009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:35:04.813669 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal
uuid: "ac74ef54542c4900954dabe583d6131c"
format_stamp: "Formatted at 2026-06-20 14:35:04 on dist-test-slave-gqzk"
server_key: "505093c55b27ccdb9c89ebe11bd4c2be"
server_key_iv: "2dfc4857a87381b4857a9ad58f3c5053"
server_key_version: "encryptionkey@0"
I20260620 14:35:04.813764 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:04.836256 27009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:04.839511 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:04.839676 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:04.846676 27009 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.125:35557
I20260620 14:35:04.846964 27076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.125:35557 every 8 connection(s)
I20260620 14:35:04.847672 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/data/info.pb
I20260620 14:35:04.850623 27077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:04.851030 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 27009
I20260620 14:35:04.851159 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-1/wal/instance
I20260620 14:35:04.851488 4329 external_mini_cluster.cc:1428] Setting key 7a7ab9ef710de6f1b6a3c1cb31fee894
I20260620 14:35:04.855391 27077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.58.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.58.124@KRBTEST.COM" created.
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:04.870719 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.857816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.4.58.126:45031)
0620 14:35:04.857993 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:04.857998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:04.858948 (+ 950us) server_negotiation.cc:407] Connection header received
0620 14:35:04.859993 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:04.859996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:04.860051 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:04.860132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:04.861489 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.862119 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.863357 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.863596 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.863685 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.866451 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:04.866454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:04.866456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:04.866484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:04.868043 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.868816 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.868820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.868821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.868924 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.869364 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.869368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.869370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.869656 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:04.869763 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:04.870430 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 14:35:04.870565 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:35:04.871536 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.858242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45031 (local address 127.0.0.1:38040)
0620 14:35:04.858759 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.858816 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:04.859750 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:04.860350 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:04.860358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:04.860812 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:04.861329 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.861343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.862259 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.862262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.863224 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.863235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.863743 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.863747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.863937 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.864489 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:04.864506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:04.866232 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:04.868172 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.868179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.868199 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.868620 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.869036 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.869039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.869042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.869240 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.869781 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:04.869789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:04.870152 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 14:35:04.870340 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:35:04.870626 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260620 14:35:04.873121 27077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:04.874756 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } attempt: 3
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:04 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
Entry for principal kudu/127.4.58.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab.
I20260620 14:35:04.880872 4329 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
/tmp/dist-test-taskCj3QH_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:39691
--webserver_interface=127.4.58.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.58.84:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:35:04.883800 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.873621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:55014)
0620 14:35:04.873773 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.873788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:04.873881 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:04.874536 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:04.874539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:04.874693 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:04.874833 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.874839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.875982 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.875986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.876535 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.876544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.877246 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.877249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.877350 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.877933 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:04.877945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:04.879466 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:04.882122 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.882125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.882127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.882362 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.882694 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.882697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.882698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.882770 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.883288 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:04.883291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:04.883376 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:35:04.883464 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:35:04.883598 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":2}
I20260620 14:35:04.883818 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.873751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.4.58.125:35557)
0620 14:35:04.874103 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:04.874112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:04.874132 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:35:04.874170 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:04.874175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:04.874398 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:04.874530 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:04.874971 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.875874 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.876642 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.877142 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.877238 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.879605 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:04.879614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:04.879630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:04.879683 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:04.882020 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.882484 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.882499 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.882502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.882589 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.882870 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.882873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.882875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.883188 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:04.883322 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:04.883557 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:35:04.883683 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":144,"threads_started":1}
I20260620 14:35:04.884840 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.875212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:55026)
0620 14:35:04.875383 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.875396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:04.875463 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:04.875825 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:04.875828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:04.875990 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:04.876105 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.876112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.876906 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.876909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.877475 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:04.877484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.878082 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.878086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:04.878193 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.878644 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:04.878659 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:04.879963 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:04.882998 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.883002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.883004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.883261 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.883567 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:04.883570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:04.883573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:04.883660 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:04.884313 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:04.884316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:04.884422 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:35:04.884528 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:35:04.884660 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:35:04.884913 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.875291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.4.58.125:35557)
0620 14:35:04.875642 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:04.875648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:04.875658 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:35:04.875683 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:04.875685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:04.875715 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:04.875819 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:04.876209 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.876796 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.877606 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:04.877953 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:04.878052 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:04.880142 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:04.880146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:04.880149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:04.880174 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:04.882874 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.883369 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.883372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.883374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.883467 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:04.883781 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:04.883784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:04.883786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:04.884019 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:04.884106 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:04.884592 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 14:35:04.884709 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":148,"threads_started":1}
I20260620 14:35:04.886191 27002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:04.886464 27077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:04.886977 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.886700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45752)
0620 14:35:04.886840 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.886897 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:35:04.887145 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.886896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45754)
0620 14:35:04.886998 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.887077 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:35:04.887444 27027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
I20260620 14:35:04.887732 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.887427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45770)
0620 14:35:04.887604 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.887651 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 14:35:04.887967 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
I20260620 14:35:04.888278 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.888074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45780)
0620 14:35:04.888194 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.888229 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 14:35:04.888402 27077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
I20260620 14:35:04.919137 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 1
I20260620 14:35:04.920050 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.919703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45784)
0620 14:35:04.919877 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.919953 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:35:04.920917 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.920634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45790)
0620 14:35:04.920794 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.920849 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:35:04.921067 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
I20260620 14:35:04.937637 27077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 1
I20260620 14:35:04.938457 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.938176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45804)
0620 14:35:04.938308 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.938377 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:35:04.939172 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.938980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45818)
0620 14:35:04.939073 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.939116 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:35:04.939303 27077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
W20260620 14:35:04.993011 27084 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:35:04.993283 27084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:35:04.993306 27084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:35:04.993355 27084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:35:04.996227 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 2
I20260620 14:35:04.997118 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.996811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45832)
0620 14:35:04.996960 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.997028 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:35:04.997509 27084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:35:04.997555 27084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:35:04.997572 27084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:35:04.997589 27084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:35:04.997617 27084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:35:04.997849 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:04.997594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45840)
0620 14:35:04.997718 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:04.997769 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:35:04.998018 27002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
I20260620 14:35:05.003252 27084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.58.84:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.58.126:45031,127.4.58.125:35557,127.4.58.124:39691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/krb5kdc/kudu_127.4.58.124.keytab
--principal=kudu/127.4.58.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.58.124:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.58.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:35:05.004359 27084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:35:05.005431 27084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:35:05.007459 27077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 2
I20260620 14:35:05.008242 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.007972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45850)
0620 14:35:05.008105 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.008166 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:35:05.009075 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.008870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45856)
0620 14:35:05.008967 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.009016 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:35:05.009196 27077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.58.124:39691: Network error: Client connection negotiation failed: client connection to 127.4.58.124:39691: connect: Connection refused (error 111)
W20260620 14:35:05.014053 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:05.014048 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:35:05.014048 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:35:05.014425 27084 server_base.cc:1034] running on GCE node
I20260620 14:35:05.014953 27084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:35:05.016026 27084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:35:05.017179 27084 hybrid_clock.cc:648] HybridClock initialized: now 1781966105017136 us; error 63 us; skew 500 ppm
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:35:05.020009 27084 init.cc:377] Logged in from keytab as kudu/127.4.58.124@KRBTEST.COM (short username kudu)
I20260620 14:35:05.021083 27084 webserver.cc:460] Webserver started at http://127.4.58.124:45405/ using document root <none> and password file <none>
I20260620 14:35:05.021659 27084 fs_manager.cc:359] Metadata directory not provided
I20260620 14:35:05.021734 27084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:35:05.021965 27084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:35:05.024091 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data/instance:
uuid: "94eae901f0a346ce8f27ca943caaf66b"
format_stamp: "Formatted at 2026-06-20 14:35:05 on dist-test-slave-gqzk"
server_key: "46c7e7f93a9310c2ab3264240668b3b7"
server_key_iv: "119ce947c22ace6ae88be0c523d664a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:05.024682 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal/instance:
uuid: "94eae901f0a346ce8f27ca943caaf66b"
format_stamp: "Formatted at 2026-06-20 14:35:05 on dist-test-slave-gqzk"
server_key: "46c7e7f93a9310c2ab3264240668b3b7"
server_key_iv: "119ce947c22ace6ae88be0c523d664a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:05.028620 27084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:35:05.031217 27100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:35:05.032528 27084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:35:05.032646 27084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data,/tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal
uuid: "94eae901f0a346ce8f27ca943caaf66b"
format_stamp: "Formatted at 2026-06-20 14:35:05 on dist-test-slave-gqzk"
server_key: "46c7e7f93a9310c2ab3264240668b3b7"
server_key_iv: "119ce947c22ace6ae88be0c523d664a1"
server_key_version: "encryptionkey@0"
I20260620 14:35:05.032769 27084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:35:05.053707 27084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:35:05.057022 27084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:35:05.057209 27084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:35:05.064148 27084 rpc_server.cc:307] RPC server started. Bound to: 127.4.58.124:39691
I20260620 14:35:05.064474 27152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.58.124:39691 every 8 connection(s)
I20260620 14:35:05.065129 27084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/data/info.pb
I20260620 14:35:05.067971 27153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:35:05.069042 4329 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu as pid 27084
I20260620 14:35:05.069175 4329 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCj3QH_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965824702348-4329-0/minicluster-data/master-2/wal/instance
I20260620 14:35:05.069437 4329 external_mini_cluster.cc:1428] Setting key 6cedcdd310b93ae881184e0e2c42999d
I20260620 14:35:05.072114 27153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:05.083482 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.074128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.4.58.126:45031)
0620 14:35:05.074257 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.074261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.074565 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:35:05.074690 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.074692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.074737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.074806 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.075513 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.076085 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.077129 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.077343 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.077427 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.079755 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.079758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.079760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.079786 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.081783 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.082223 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.082227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.082229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.082328 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.082601 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.082604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.082605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.082809 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.082902 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.083217 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.083342 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:35:05.083840 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.070772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.4.58.124:39691)
0620 14:35:05.071169 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.071185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.071229 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:35:05.071856 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.071873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.072237 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.072541 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.073521 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.074847 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.075820 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.076441 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.076591 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.079196 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.079204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.079218 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.079244 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.080647 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.081119 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.081124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.081129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.081301 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.081585 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.081588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.081590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.082088 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.082276 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.082543 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.082852 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260620 14:35:05.083858 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.073999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.126:45031 (local address 127.0.0.1:38046)
0620 14:35:05.074412 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.074474 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:05.074604 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:05.074869 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:05.074874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:05.075253 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:05.075388 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.075395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.076194 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.076197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.077020 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.077029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.077476 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.077481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.077571 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.078090 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:05.078111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:05.079620 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:05.081902 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.081910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.081915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.082122 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.082420 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.082423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.082424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.082512 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.082913 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:05.082918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:05.083020 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:35:05.083133 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:35:05.083269 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260620 14:35:05.085176 27153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.126@KRBTEST.COM
I20260620 14:35:05.093299 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.085695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.125:35557 (local address 127.0.0.1:55036)
0620 14:35:05.085882 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.085894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:05.085965 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:05.086191 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:05.086194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:05.086375 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:05.086484 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.086489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.087271 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.087274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.087851 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.087862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.088323 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.088327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.088429 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.088970 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:05.088987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:05.090302 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:05.091894 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.091898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.091900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.092128 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.092381 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.092384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.092385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.092477 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.092837 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:05.092840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:05.092941 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 14:35:05.093021 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:35:05.093125 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 14:35:05.093349 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.085792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.4.58.125:35557)
0620 14:35:05.085939 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.085944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.085959 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:35:05.086036 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.086039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.086089 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.086161 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.086580 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.087152 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.087962 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.088214 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.088298 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.090439 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.090442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.090444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.090468 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.091784 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.092219 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.092222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.092223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.092297 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.092556 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.092557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.092559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.092718 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.092879 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.093092 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.093220 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:35:05.094398 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.086294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.4.58.126:45031)
0620 14:35:05.086403 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.086407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.086418 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:05.086522 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.086525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.086568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.086639 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.087253 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.087726 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.088617 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.088915 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.088998 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.091313 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.091317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.091320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.091348 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.092727 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.093172 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.093176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.093178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.093278 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.093570 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.093574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.093576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.093794 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.094021 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.094135 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.094243 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:35:05.094575 27153 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:35:05.097152 27002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 3
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.124@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:35:05.100329 27077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } attempt: 3
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.58.125@KRBTEST.COM
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.126@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:35:05.103533 27157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.095078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45870)
0620 14:35:05.095191 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.095208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:05.095286 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:05.095596 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:05.095599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:05.095769 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:05.095841 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.095846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.096586 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.096590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.097325 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.097338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.098020 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.098024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.098128 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.098545 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:05.098559 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:05.099895 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:05.101724 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.101727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.101729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.102134 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.102387 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.102390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.102392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.102465 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.103035 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:05.103038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:05.103130 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:35:05.103221 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:35:05.103354 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3,"spinlock_wait_cycles":1792}
I20260620 14:35:05.104030 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.095299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.4.58.124:39691)
0620 14:35:05.095417 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.095421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.095432 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:05.095464 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.095467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.095509 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.095585 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.095946 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.096477 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.097457 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.097892 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.098021 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.100024 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.100028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.100031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.100057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.101636 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.102231 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.102234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.102236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.102304 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.102541 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.102567 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.102568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.102921 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.103003 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.103768 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.103889 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":58}
I20260620 14:35:05.105973 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.095589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.4.58.125:35557)
0620 14:35:05.095691 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.095695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.095706 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:05.095826 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.095829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.095868 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.095930 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.096487 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.097133 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.098543 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.098864 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.098971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.101699 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.101703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.101705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.101730 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.103397 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.103848 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.103852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.103854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.103948 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.104557 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.104561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.104562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.104806 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.105009 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.105587 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.105753 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:35:05.106832 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.097511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45876)
0620 14:35:05.097606 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.097619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:05.097689 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:05.098224 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:05.098227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:05.098402 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:05.098490 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.098496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.099779 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.099782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.100431 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.100443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.101112 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.101116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.101199 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.101576 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:05.101590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:05.102916 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:05.105044 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.105047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.105049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.105267 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.105687 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.105691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.105694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.105819 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.106362 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:05.106377 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:05.106471 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:35:05.106551 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:35:05.106680 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:35:05.107613 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.097677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.4.58.124:39691)
0620 14:35:05.098025 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.098034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.098045 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:35:05.098073 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.098075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.098116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.098200 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.098594 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.099623 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.100605 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.100980 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.101451 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.103062 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.103066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.103068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.103105 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.104936 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.105414 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.105417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.105419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.105534 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.105965 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.105968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.105970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.106250 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.107264 (+ 1014us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.107334 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.107455 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":5888,"thread_start_us":122,"threads_started":1}
Jun 20 14:35:05 dist-test-slave-gqzk krb5kdc[26921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.58.125@KRBTEST.COM for kudu/127.4.58.124@KRBTEST.COM
I20260620 14:35:05.108901 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: Bootstrap starting.
I20260620 14:35:05.110538 27002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: Bootstrap starting.
I20260620 14:35:05.111706 27153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:05.112196 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.100709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.58.124:39691 (local address 127.0.0.1:45884)
0620 14:35:05.101203 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 14:35:05.101217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:35:05.101285 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:35:05.102387 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:35:05.102391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:35:05.102785 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:35:05.102896 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.102904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.104736 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.104739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.105461 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:35:05.105471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.106155 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.106158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:35:05.106241 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.106831 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:35:05.106845 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:35:05.108384 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:35:05.110662 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.110665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.110667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.110910 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.111195 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:35:05.111198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:35:05.111200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:35:05.111288 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:35:05.111692 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:35:05.111695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:35:05.111800 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:35:05.111908 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:35:05.112047 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420}
I20260620 14:35:05.112263 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.101611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.4.58.124:39691)
0620 14:35:05.101915 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.101922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.101935 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:35:05.102190 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.102193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.102236 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.103002 (+ 766us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.103113 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.104046 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.105587 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.105980 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.108874 (+ 2894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.108979 (+ 105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.108983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.108985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.109001 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.110549 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.111022 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.111025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.111027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.111107 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.111387 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.111390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.111392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.111591 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.111686 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.112001 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.112108 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260620 14:35:05.112550 27153 log.cc:824] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:05.113137 27002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:05.113974 27002 log.cc:824] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:05.114166 4329 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:35:05.115271 27077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: Bootstrap starting.
I20260620 14:35:05.116168 27153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: No bootstrap required, opened a new log
I20260620 14:35:05.115352 26976 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "992adbcd6adf4d2981356b5e7c21ef6d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38056
I20260620 14:35:05.117990 27077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: Neither blocks nor log segments found. Creating new log.
I20260620 14:35:05.118816 27077 log.cc:824] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:35:05.119877 27153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.120082 27153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94eae901f0a346ce8f27ca943caaf66b, State: Initialized, Role: FOLLOWER
I20260620 14:35:05.120412 27153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.120960 27077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: No bootstrap required, opened a new log
I20260620 14:35:05.120921 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.121059 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.121785 27153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:05.124874 27077 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.125111 27077 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac74ef54542c4900954dabe583d6131c, State: Initialized, Role: FOLLOWER
I20260620 14:35:05.125442 27077 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.125658 27002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: No bootstrap required, opened a new log
I20260620 14:35:05.126909 27077 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:35:05.127105 27174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:05.127211 27174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:05.127512 27176 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.127719 27176 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.128638 27002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.129701 27002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992adbcd6adf4d2981356b5e7c21ef6d, State: Initialized, Role: FOLLOWER
I20260620 14:35:05.129559 27084 master_runner.cc:186] Error getting master registration for 127.4.58.126:45031: OK, instance_id { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" instance_seqno: 1781966104632127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:05.132725 27084 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:35:05.132815 27187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:05.132892 27187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:05.131034 27009 master_runner.cc:186] Error getting master registration for 127.4.58.126:45031: OK, instance_id { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" instance_seqno: 1781966104632127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:35:05.134289 27009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:05.134631 27002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.135228 27189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.135474 27189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.136133 27002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:35:05.136209 26976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:35:05.136297 26976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:05.137732 26976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
I20260620 14:35:05.138751 26976 leader_election.cc:288] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [CANDIDATE]: Term 1 election: Requested vote from peers ac74ef54542c4900954dabe583d6131c (127.4.58.125:35557), 94eae901f0a346ce8f27ca943caaf66b (127.4.58.124:39691)
I20260620 14:35:05.139613 27127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94eae901f0a346ce8f27ca943caaf66b"
I20260620 14:35:05.139894 27127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:05.141346 27127 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992adbcd6adf4d2981356b5e7c21ef6d in term 1.
I20260620 14:35:05.141390 27050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac74ef54542c4900954dabe583d6131c"
I20260620 14:35:05.141644 27050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:35:05.142190 26952 leader_election.cc:302] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94eae901f0a346ce8f27ca943caaf66b, 992adbcd6adf4d2981356b5e7c21ef6d; no voters:
I20260620 14:35:05.142984 27050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992adbcd6adf4d2981356b5e7c21ef6d in term 1.
I20260620 14:35:05.143400 27189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:35:05.143599 27189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [term 1 LEADER]: Becoming Leader. State: Replica: 992adbcd6adf4d2981356b5e7c21ef6d, State: Running, Role: LEADER
I20260620 14:35:05.143936 27189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } }
W20260620 14:35:05.145229 27200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:35:05.145332 27200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:35:05.146029 27189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992adbcd6adf4d2981356b5e7c21ef6d. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.146135 27189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:05.148137 26934 master_runner.cc:263] No leader master found from master 992adbcd6adf4d2981356b5e7c21ef6d
I20260620 14:35:05.148197 26934 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:35:05.158782 27209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:35:05.159822 27209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:35:05.168035 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.146468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.4.58.125:35557)
0620 14:35:05.146575 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.146580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.157052 (+ 10472us) server_negotiation.cc:407] Connection header received
0620 14:35:05.157234 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.157238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.157292 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.157343 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.158300 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.159010 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.160270 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.160573 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.160650 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.164422 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.164426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.164429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.164461 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.166209 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.166632 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.166636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.166638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.166731 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.167050 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.167053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.167055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.167282 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.167381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.167719 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.167848 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:35:05.170814 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.145773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.4.58.126:45031)
0620 14:35:05.145929 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.145934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.153070 (+ 7136us) server_negotiation.cc:407] Connection header received
0620 14:35:05.153292 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.153296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.153348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.153396 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.161945 (+ 8549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.162911 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.164080 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.164438 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.164511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.166897 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.166901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.166904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.166933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.168952 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.169504 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.169507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.169509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.169593 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.169870 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.169873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.169875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.170081 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.170192 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.170513 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.170641 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:35:05.173841 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:05.146909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.4.58.124:39691)
0620 14:35:05.147018 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.147023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.162221 (+ 15198us) server_negotiation.cc:407] Connection header received
0620 14:35:05.162405 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.162408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.162461 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.162509 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.163480 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.164260 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.165346 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:05.165759 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:05.165837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:05.167514 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:35:05.167517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:35:05.167520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:35:05.167548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:35:05.169013 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.172429 (+ 3416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.172433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.172435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.172535 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:35:05.172852 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:35:05.172856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:35:05.172857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:35:05.173073 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:35:05.173183 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:05.173521 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:35:05.173667 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:35:05.177093 27127 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [term 1 FOLLOWER]: Refusing update from remote peer 992adbcd6adf4d2981356b5e7c21ef6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:05.177714 27050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [term 1 FOLLOWER]: Refusing update from remote peer 992adbcd6adf4d2981356b5e7c21ef6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:35:05.177969 27206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:05.178411 27206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:35:05.184330 27176 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992adbcd6adf4d2981356b5e7c21ef6d. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.184468 27176 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.188819 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992adbcd6adf4d2981356b5e7c21ef6d. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.188972 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.193688 27206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.193842 27206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:05.194455 27176 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.194572 27176 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.196836 27213 mvcc.cc:205] Tried to move back new op lower bound from 7298933166797815808 to 7298933166673174528. Current Snapshot: MvccSnapshot[applied={T|T < 7298933166797815808 or (T in {7298933166797815808})}]
I20260620 14:35:05.209172 27206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.209326 27206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d [sys.catalog]: This master's current role is: LEADER
I20260620 14:35:05.210431 27163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "992adbcd6adf4d2981356b5e7c21ef6d" member_type: VOTER last_known_addr { host: "127.4.58.126" port: 45031 } } peers { permanent_uuid: "ac74ef54542c4900954dabe583d6131c" member_type: VOTER last_known_addr { host: "127.4.58.125" port: 35557 } } peers { permanent_uuid: "94eae901f0a346ce8f27ca943caaf66b" member_type: VOTER last_known_addr { host: "127.4.58.124" port: 39691 } } }
I20260620 14:35:05.210566 27163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:35:05.211230 27209 catalog_manager.cc:1347] Generated new cluster ID: ef468a86f4b44920990bfa4e65ec2990
I20260620 14:35:05.211298 27209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:35:05.227010 27215 mvcc.cc:205] Tried to move back new op lower bound from 7298933166797815808 to 7298933166673174528. Current Snapshot: MvccSnapshot[applied={T|T < 7298933166797815808 or (T in {7298933166797815808})}]
I20260620 14:35:05.235718 27209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:35:05.237368 27209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:35:05.255510 27209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 992adbcd6adf4d2981356b5e7c21ef6d: Generated new TSK 0
I20260620 14:35:05.258875 27209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:35:05.297778 27005 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.293986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.4.58.126:45031)
0620 14:35:05.294133 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.294138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.296460 (+ 2322us) server_negotiation.cc:407] Connection header received
0620 14:35:05.296733 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.296738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.296838 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.296892 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.297616 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38070: BlockingRecv error: recv got EOF from 127.0.0.1:38070 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 14:35:05.299863 27159 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.294954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.4.58.124:39691)
0620 14:35:05.295079 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.295083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.298247 (+ 3164us) server_negotiation.cc:407] Connection header received
0620 14:35:05.298948 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.298952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.299009 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.299063 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.299710 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45898: BlockingRecv error: recv got EOF from 127.0.0.1:45898 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:35:05.300942 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.295778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.4.58.125:35557)
0620 14:35:05.295903 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.295908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.298644 (+ 2736us) server_negotiation.cc:407] Connection header received
0620 14:35:05.300138 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.300141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.300217 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.300276 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.300822 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55060: BlockingRecv error: recv got EOF from 127.0.0.1:55060 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:35:05.306550 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.304491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.4.58.125:35557)
0620 14:35:05.304613 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.304618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.305313 (+ 695us) server_negotiation.cc:407] Connection header received
0620 14:35:05.305563 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.305567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.305652 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.305714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.306435 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55064: BlockingRecv error: recv got EOF from 127.0.0.1:55064 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:35:05.312875 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.311120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.4.58.125:35557)
0620 14:35:05.311242 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.311246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.311916 (+ 670us) server_negotiation.cc:407] Connection header received
0620 14:35:05.312109 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.312112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.312163 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.312224 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.312740 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55076: BlockingRecv error: recv got EOF from 127.0.0.1:55076 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:35:05.319517 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.317521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.4.58.125:35557)
0620 14:35:05.317634 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.317638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.318468 (+ 830us) server_negotiation.cc:407] Connection header received
0620 14:35:05.318695 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.318699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.318755 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.318817 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.319394 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55092: BlockingRecv error: recv got EOF from 127.0.0.1:55092 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 14:35:05.327793 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.326030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.4.58.125:35557)
0620 14:35:05.326181 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.326185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.326801 (+ 616us) server_negotiation.cc:407] Connection header received
0620 14:35:05.327009 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.327012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.327065 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.327113 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.327671 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55106: BlockingRecv error: recv got EOF from 127.0.0.1:55106 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:35:05.339780 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.338490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.4.58.125:35557)
0620 14:35:05.338616 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.338621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.338909 (+ 288us) server_negotiation.cc:407] Connection header received
0620 14:35:05.339052 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.339055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.339118 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.339186 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.339660 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55118: BlockingRecv error: recv got EOF from 127.0.0.1:55118 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:35:05.358971 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.357816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.4.58.125:35557)
0620 14:35:05.357967 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.357971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.358076 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:35:05.358205 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.358209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.358266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.358370 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.358839 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55124: BlockingRecv error: recv got EOF from 127.0.0.1:55124 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:35:05.394279 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.393152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.4.58.125:35557)
0620 14:35:05.393330 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.393335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.393431 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:35:05.393585 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.393588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.393636 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.393709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.394177 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55126: BlockingRecv error: recv got EOF from 127.0.0.1:55126 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 14:35:05.461556 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.460459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.4.58.125:35557)
0620 14:35:05.460594 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.460599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.460671 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:35:05.460863 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.460866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.460914 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.460984 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.461458 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55132: BlockingRecv error: recv got EOF from 127.0.0.1:55132 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 14:35:05.592888 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.591686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.4.58.125:35557)
0620 14:35:05.591856 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.591861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.591962 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:35:05.592121 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.592124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.592173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.592257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.592746 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55148: BlockingRecv error: recv got EOF from 127.0.0.1:55148 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 14:35:05.852543 27083 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:35:05.851295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.4.58.125:35557)
0620 14:35:05.851432 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:05.851437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:05.851619 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:35:05.851747 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:05.851749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:05.851801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:05.851870 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:35:05.852429 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52422: BlockingRecv error: recv got EOF from 127.0.0.1:52422 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:35:06.128657 27174 catalog_manager.cc:1259] Loaded cluster ID: ef468a86f4b44920990bfa4e65ec2990
I20260620 14:35:06.128777 27174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: loading cluster ID for follower catalog manager: success
I20260620 14:35:06.130511 27174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: acquiring CA information for follower catalog manager: success
I20260620 14:35:06.131688 27174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94eae901f0a346ce8f27ca943caaf66b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:06.134145 27187 catalog_manager.cc:1259] Loaded cluster ID: ef468a86f4b44920990bfa4e65ec2990
I20260620 14:35:06.134222 27187 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: loading cluster ID for follower catalog manager: success
I20260620 14:35:06.136296 27187 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: acquiring CA information for follower catalog manager: success
I20260620 14:35:06.137627 27187 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac74ef54542c4900954dabe583d6131c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:35:06.371115 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.367085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.4.58.125:35557)
0620 14:35:06.367355 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:06.367358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:06.367407 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:35:06.367522 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:06.367524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:06.367653 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:06.367766 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:06.368068 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.368625 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.369556 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.369793 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.369874 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:06.370248 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:06.370718 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:06.370829 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:06.370905 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:35:06.370972 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260620 14:35:06.378582 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.374446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.4.58.124:39691)
0620 14:35:06.374735 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:06.374740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:06.374761 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:35:06.374986 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:06.374988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:06.375133 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:06.375270 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:06.375522 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.376063 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.376979 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.377216 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.377320 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:06.377680 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:06.378142 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:06.378270 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:06.378350 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:35:06.378414 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":67,"threads_started":1}
I20260620 14:35:06.386096 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.381866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.4.58.124:39691)
0620 14:35:06.381979 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:06.381983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:06.382451 (+ 468us) server_negotiation.cc:407] Connection header received
0620 14:35:06.382591 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:06.382594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:06.382640 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:06.382704 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:06.383006 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.383819 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.385034 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.385276 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.385375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:06.385629 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:06.385703 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:06.385758 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:06.385888 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:35:06.385950 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:35:06.386289 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.381874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.4.58.125:35557)
0620 14:35:06.382030 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:06.382034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:06.382343 (+ 309us) server_negotiation.cc:407] Connection header received
0620 14:35:06.382529 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:06.382532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:06.382592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:06.382658 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:06.382903 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.383493 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.384579 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.384946 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.385043 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:06.385820 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:06.385898 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:06.385970 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:06.386104 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:35:06.386165 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:35:06.386466 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:35:06.381993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.4.58.126:45031)
0620 14:35:06.382223 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:35:06.382228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:35:06.382291 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:35:06.382569 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:35:06.382573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:35:06.382750 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:35:06.382867 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:35:06.383061 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.383851 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.384950 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:35:06.385223 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:35:06.385307 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:35:06.385652 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:35:06.386014 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:35:06.386098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:35:06.386242 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:35:06.386322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260620 14:35:06.390020 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 26934
I20260620 14:35:06.395869 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 27009
I20260620 14:35:06.401499 4329 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCj3QH_/build/debug/bin/kudu with pid 27084
2026-06-20T14:35:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3025 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6028 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281690 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST